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ABSTRACT 

Exercise  Black  Skies  2008  (EBS08)  was  a  simulation  exercise  conducted  in  the  weeks  prior  to  the  live  air 
combat  training  exercise,  Pitch  Black  2008  (PB08).  During  EBS08,  a  Royal  Australian  Air  Force  (RAAF)  air 
battl  e  management  (ABM)  team  cond  ucted  a  seri  es  of  vi  gnettes  desi  gned  to  prepare  them  for  thei  r  tacti  cal 
command  and  control  rolein  PB08.  A  broad  goal  of  EBS08wasto  providean  environment  within  which  a 
number  of  new  simulation  tools  and  training  techniques  could  be  evaluated  and  developed  for  future 
implementation  within  the  Royal  Australian  Air  Force  (RAAF).  Thetools  and  techniques  evaluated  were: 
the  Air  Defence  Ground  Environment  Simulator  (ADGESIM),  Toteboard,  Air  Warfare  Assessment  and 
Review  tool,  After  Action  Review  tools  and  theTeam-Dimensional  Training  framework.  Theseevaluations, 
which  form  the  basis  or  this  report,  were  informed  by  the  observations  of  human  factors  scientists  and 
interviewswith  the  RAAF  participants.  Participants  provided  extensive  feed  back  on  each  tool,  highlighting 
shortcomings  and  providing  valuable  suggestions  for  improvement.  Thisfeedback  will  guide  the  further 
devel  opment  of  these toolsfor  i  mplementati  on  i  n  future  research  exercises  and  i  n  RA  A  F  trai  ni  ng  programs. 
Overall,  the  participants  found  EBS08  extremely  valuable  and  were  satisfied  with  the  quality  of  training 
they  received.  The  simulation  tools  and  training  techniques  assisted  in  promoting  the  quality  of  this 
experience  and  prompted  participants  to  consider  methods  by  which  the  current  RAAF  approach  to 
training  could  be  improved. 
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Exercise  Black  Skies  2008:  Enhancing  LiveTraining 
Through  Virtual  Preparation 


Part  Two:  An  Evaluation  of  Tools  and  Techniques 


Executi  ve  Su  mmary 

Exerci  se  Bl  ack  Ski  es  2008  (E  BS08)  was  a  si  mu  I  ati  on  exerci  se  conducted  under  the  Defence 
Science  and  Technology  Organisation's  (DSTO's)  Support  to  Air  Force  Readiness  and 
Renewal  task  (AIR  07/  232)  and  Project  Arrangement  PA-AF-0025  "Distributed  Mission 
Training  Effectiveness  Research"  with  Air  Force  Research  Laboratories  in  the  United 
States.  EBS08  was  envisaged  as  a  preparatory  exercise  for  a  Royal  Australian  Air  Force 
(RAAF)  air  battle  management  (ABM)  team  who  were  to  be  involved  in  Exercise  Pitch 
Black  2008  (PB08)  and  was  conducted  in  theweeks  prior  to  the  commencement  of  PB08. 
Pitch  Black  is  a  biennial,  live  air  combat  training  exercise,  hosted  by  the  RAAF  and 
involving  participants  from  a  number  of  foreign  military  forces.  EBS08  was  designed  to 
resemble  PB08  as  closely  as  possible  in  terms  of  scenario  vignettes,  orders  of  battle, 
airspace,  procedures  and  tactics.  The  role  of  ABM  teams  in  both  exercises  was  to  provide 
command  and  control  of  the  Red  Force  air  assets.  At  EBS08,  a  White  Force,  including 
RA  A  F  si  mu  I  ator  operators  (SI  M  O  Ps)  and  a  White  Force  mi  ssi  on  director,  were  tasked  with 
generati  ng  and  control  I  i  ng  thesyntheti  c  Red  and  Bl  ue  Forces  i  n  a  manner  consi  stent  with 
thescaleand  complexity  of  the  PB08  scenario. 

Thi  s  report  i  s  the  second  i  n  a  two-part  publ  i  cati  on  seri  es  add  ressi  ng  the  research  outcomes 
of  EBS08.  EBS08  had  two  broad  goals,  the  first  of  which  is  discussed  in  Part  One  of  the 
series  (Shanahan,  etal.,  in  preparation)  and  the  second  of  which  will  bediscussed  inthe 
present  report.  The  first  goal  of  the  exerci  sew  as  to  examine  the  benefits  of  providing  the 
R  A  A  F  A  B  M  team  w  i  th  a  sy  nth  eti  c  trai  n  i  n  g  en  vi  ron  ment  w  i  th  i  n  w  h  i  ch  to  p  repare  for  th  ei  r 
tactical  command  and  control  role  in  PB08.  Shanahan,  et  al.  therefore  discusses  the 
effectiveness  of  EBS08  as  a  preparatory  exercise.  The  second  goal  was  to  provide  an 
environment  within  which  a  number  of  new  si  mu  I  ati  on  tools,  decision  aids,  performance 
measurement  toolsand  training  techniques  could  be  evaluated  and  developed  for  future 
i  mpl  ementati  on  w  i  thi  n  the  RAAF.  Thetool  s  and  techni  ques  tri  al  I  ed  i  n  E  BS08  were  the  A  i  r 
Defence  Ground  Environment  Simulator  ( ADGESIM )  suite  of  simulation  tools,  the 
Toteboard,  the  Air  Warfare  Assessment  and  Review  ( AWAR )  tool,  a  number  of  After 
Action  Review  (AAR)  toolsand  the  Team- Dimensional  Training  (IDF)  framework.  The 
present  report  provides  an  evaluation  of  these  tools  and  techniques  in  terms  of  their 
potential  for  use  in  a  RAAF  training  context  and  suggests  methods  by  which  they  may  be 
improved.  This  evaluation  was  based  on  i  nformati on  gleaned  from  structured  discussions 
with  the  ABM  team  and  SIMOPs  and  the  general  observations  of  DSTO  human  factors 
scientists. 


The/\  D  G  E  SI  M  su  i  te  of  tool  s  consi  sts  of  the  D  /  S  V  0  X  rad  i  o  si  mu  I  ator,  the  P  i  I  ot  Si  mu  I  ator 
I  interface (PSI ),  theSensorLIN  K  simulated  sensor  gateway,  theSimM  an  simulator  manager, 
theTactical  After  Action  Review  for  Distributed  Interactive  Simulation  ( TAARDIS )  and 
the  1/1/  or  I  dV  I EW  three-d  i  mensi  onal  vi  sual  i  sati  on  tool .  These  tool  s  are  cu  rrentl  y  i  n  use  at  the 
Survei  1 1  ance  and  Control  Trai  ni  ng  U  nit  (SACTU )  i  n  Wi  1 1  i  amtown,  N  ew  South  Wal  es,  and 
the  School  of  A  i  r  Warfare  (SAW)  i  n  Sal  e,  Vi  ctori  a.  Pri  or  to  EBS08,  th  eA  DG  ESI  M  tool  s  were 
mod  i  f  i  ed  to  red  uce  SI  M  O  P  w  or  kl  oad  i  n  I  arge  scenari  os  and  to  i  mp  rove  the  consi  stency  of 
behaviour  representation.  The  version  of  ADGESIM  trialled  at  EBS08  was  considered  a 
significant  improvement  over  the  version  in  use  at  SACTU  and  SAW.  The  RAAF 
parti ci  pants  were  impressed  with  the  potential  of  the  new  system  to  run  effectively  with 
fewer  operators  than  the  previous  version. 

The  Toteboard  is  a  customised  electronic  spreadsheet  that  was  developed  at  SACTU  to 
provide  ABM  teams  with  situation  awareness  on  mission-specific  aspects,  including 
aircraft  fuel  states,  alert  states,  weapon  loadouts,  assigned  airspace  and  radio  frequency, 
tanker  offload  levels,  endurance  estimates,  and  aircraft  turn-around  timings.  The  ABM 
teamfeltthattherofeboard  provided  all  information  required  during  mission  execution.  It 
also  provided  the  additional  benefit  of  facilitating  coordination  between  the  tactical 
control  centre  and  the  White  Force  control  centre. 

The  goal  of  TDT  is  to  improve  team  performance  in  four  areas,  namely  information 
exchange,  communication,  supporting  behaviour  and  initiative  leadership.  The  RAAF 
parti  ci  pants  reported  that  these  teamwork  constructs  arei  mpl  i  cit  i  n  current  RAAF  trai  ni  ng 
and  performance  assessment,  and  that  addressing  them  explicitly  during  debrief  is 
unlikely  to  provide  additional  training  benefit.  During  EBS08,  TDT  was  only  partially 
implemented  and  therefore  conclusions  cannot  be  drawn  regarding  the  utility  of  the 
approach.  FI  ad  TDT  been  implemented  in  its  entirety  however,  the  rel  uctance  of  RAAF 
participants  to  accept  and  implement  the  approach  may  have  impacted  negativelyon  its 
effecti  veness.  Thi  s  hi  ghl  i  ghts  the  i  importance  of  gai  ni  ng  the  RA  A  F  parti  ci  pants'  acceptance 
of  any  teamwork  tool  incorporated  in  future  exercises,  either  through  a  thorough 
explanation  of  thebenefitsof  thechosen  approach  or  by  modifying  an  approach  to  bring  it 
into  linewith  the  culture  of  the  RAAF. 

The  AWAR  tool,  created  at  DSTO,  was  designed  to  promote  the  consistency  and 
objectivity  of  assessment  rati  ngs  and  to  providean  assessor  with  focus  areas  around  which 
tostructuredebriefs.The^  WAR  tool,  incorporating  the  ABM  DefensiveCounter  Airteam 
goal  hierarchy  offered  a  marked  improvement  on  current  RAAF  methodology  and  tools, 
i  n  terms  of  the  breadth,  depth  and  consi  stency  of  rati  ngs.  Parti  ci  pants  felt  that  the  team 
goal  hierarchy  would  beuseful  for  designing  basic  course  scenari  os  at  SACTU  and  SAW. 
TheABM  team  found  the  visual  layout  of  the  debrief  user  interface  intuitive.  The  scores 
and  comments  relating  to  each  sub-goal  were  extremely  useful,  as  they  provided  theteam 
with  an  understanding  of  their  current  performance,  showed  how  much  room  existed  for 
improvement,  and  provided  instructions  for  improving. 

The  AAR  tools  were  to  be  used  by  the  assessor  to  visually  and  aurally  support  the 
debri  efi  ng  comments  he  made  duri  ng  the  A  A  Rs.  The  RA  A  F  parti  ci  pants  found  the  A  A  R 
tools  very  effecti  vein  minimi  sing  theamount  of  preparation  required  between  theend  of 
a  mission  and  the  beginning  of  the  task  performance  debrief.  The  task  assessor  also  felt 
thattheA  A  R  tool  s  i  mproved  theti  meeffici  ency  of  thedebri  ef  process  itself  by  faci  I  itati  ng 
the  brevity  and  sharpness  of  learning  points.  The  playback  of  video  and  audio  was 
considered  an  extremely  useful  method  of  illustrating  important  learning  points. 


Overall,  the  participants  found  EBS08  to  be  an  extremely  valuable  exercise  and  were 
satisfied  with  the  quality  of  the  preparation  they  received  for  PB08.  The  simulation  tools 
and  training  techniques  evaluated  in  the  exercise  assisted  in  promoting  thequality  of  this 
experi  ence  and  prompted  parti  ci  pants  to  consi  der  methods  by  w  hi  ch  thecurrent  approach 
of  the  RA  A  F  to  trai  ni  ng  cou  I  d  be  i  mproved .  Parti  ci  pants  provi  ded  extensi  ve  feed  back  on 
each  tool,  highlighting  shortcomings  and  providing  valuable  suggestions  for 
improvement.  This  feedback  will  guide  the  further  development  of  these  tools  for 
implementation  in  future  research  exercises  and  RAAF  training  programs. 
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1.  Introduction 

Exercise  Black  Skies  2008  (EBS08)  was  a  simulation  exercise  conducted  under  the  Defence 
Science  and  Technology  Organisation's  (DSTO)  Support  to  Air  Force  Readiness  and  Renewal 
task  (AIR  07/  232)  and  Project  Arrangement  PA-AF-0025  "Distributed  Mission  Training 
Effecti  veness  Research  "  w  i  th  A  i  r  Force  Research  L  aboratori  es  i  n  the  U  n  i  ted  States.  E  BS08  w  as 
envisaged  as  a  preparatory  exercise  for  a  Royal  Australian  Air  Force  (RAAF)  air  battle 
management  (ABM )  team  who  were  to  be  involved  in  Exercise  Pitch  Black  2008  (PB08)  and 
w  as  cond  u  cted  i  n  th  e  w  eeks  p  r  i  or  to  th  e  commen  cement  of  P  B08.  P  i  tch  Bl  ack  isabiennialjive 
air  combat  training  exercise,  hosted  by  the  RAAF  and  involving  parti  ci  pants  fromanumber  of 
forei  gn  mi  I  i  tary  forces.  E  BS08  w  as  desi  gned  to  resembl  e  P  B08  as  cl  osel  y  as  possi  bl  e  i  n  terms  of 
scenario  vignettes,  orders  of  battle  (OR BAT),  airspace,  procedures  and  tactics.  EBS08  can  be 
cl  assi  f  i  ed  as  an '  exerci  se  workup',  as  definedinthe  conti  nuum  of  simul  ated  trai  n  i  n  g  events  i  n 
A  ppendi  x  A .  The  rol  eof  A  BM  teams  i  n  both  exerci  ses  was  to  provi  de  command  and  control  of 
the  Red  Force  air  assets.  At  EBS08,  a  White  Force,  including  RAAF  simulator  operators 
(SIMOPs)  and  a  White  Force  mission  director  (WFMD)  on  contract  from  Milskil  Integrated 
Defence  Solutions  (Cooks  FH i II,  NSW),  were  tasked  with  generating  and  controlling  the 
synthetic  Red  and  Blue  Forces  in  a  manner  consistent  with  the  scale  and  complexity  of  the 
PB08  scenarios.  TheWFMD  had  been  involved  i  n  the  development  of  the  PB08  scenario  and 
was  to  play  a  key  White  Force  role  in  PB08. 

EBS08had  two  broad  goals.  These  goals  are  discussed  along  with  relevant  research  outcomes 
in  a  two-part  publication  series.  The  first  goal,  discussed  in  Shanahan,  etal.  (in  preparation), 
was  to  examine  the  benefits  of  providing  a  RAAF  ABM  team  from  41  Wing  (41WG) 
Surveillance  and  Response  G  rou  p  (SRG)  with  a  synthetic  training  environmentwithin  which 
to  prepare  for  PB08.  The  second  goal,  discussed  in  the  present  report,  was  to  provide  an 
environment  within  which  a  number  of  new  simulation  tools,  decision  aids,  performance 
measurement  tool  s  and  trai  ni  ng  techni  ques  could  be  eval  uated  i  n  terms  of  thei  r  utility  within 
RAAF  training  programs  and  the  methods  by  which  they  could  be  improved.  The  tools  and 
techniques  evaluated  in  the  exercise,  and  thus  discussed  in  this  report,  are  the  Air  Defence 
Ground  Environment  Simulator  ( ADGESIM )  suite  of  simulation  tools,  the  Tofeboard,  the  Air 
WarfareAssessmentand  Review  ( A 1/1/4  R)  tool,  a  number  of  After  Action  Review  (AAR)tools 
and  theT eam-Di  mensional  T rai  ni  ng  (T D  T)  framework.  The  envi  ronment  produced  through 
the  use  of  these  tools  and  techniques  represented  an  evolved  approach  to  training  over 
methods  currently  implemented  by  the  RAAF. 

The  body  of  this  report  is  composed  of  two  sections.  The  first  section  considers  aspects 
relating  to  the  planning  and  conduct  of  EBS08.  It  provides  a  description  of  the  personnel 
i  nvol  ved  i  n  the  exerci  se  and  the  rol  es  they  pi  ayed ,  and  of  scenari  o  management  and  exerci  se 
workfl  ow.  Theexerci  sefaci  I  ity,  i  nfrastru  dure  and  tools  arethen  descri  bed  i  n  detai  I ,  outl  i  ni  ng 
themethodsby  which  they  were  implemented  during  EBS08  and  the  potential  improvements 
they  offer  over  approaches  currently  employed  by  theRAAF.Thesecond  section  discusses  the 
utility  of  each  tool  or  method,  potential  modifications  and  potential  applications  in  RAAF 
training  programs.  Thesediscussionsarebased  on  information  gleaned  from  interviews  with 
the  RAAF  participants  during  the  exercise  and  the  observations  of  DSTO  human  factors 
scientists.  The  discussion  of  each  tool  concludes  with  a  list  of  recommendations.  A 
consol  i  dated  I  ist  of  these  recommendati ons  i s  avai  I abl  e  i  n  A ppend i x  B. 
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2.  M  ethods  and  tools 

2.1  Exercise  planning  and  execution 

Thefol  I  owing  subsections  will  providean  overview  of  the  scenarios  employed  atEBS08,the 
parti  ci  pant  rol  es  and  responsi  bi  I  i ti  es  the  rol  es  of  exerci  se  staff.  The  sched u I  e  for  the  week  i  s 
outlined,  along  with  the  system  malfunctions  that  occurred  and  their  effect  on  the  exercise 
schedule.  The  procedure  is  then  described,  including  data  collection  and  debriefing 
procedures. 

2.1.1  EBS08  Scenarios 

A  major  goal  of  EBS08  was  to  prepare  a  41WG  ABM  team  for  their  Red  Force  tactical 
command  and  control  (C2)  rolein  PB08.  The  scenario  vignettes  used  in  EBS08  were  identical 
to  those  proposed  for  PB08,  in  an  effort  to  promote  the  transfer  of  training  benefits  from  the 
simulated  exercise  to  the  live  exercise.  The  exercise  scenario  developed  for  PB08,  and  thus 
used  also  in  EBS08,  was  designed  to  providethe  Blue  Force  military  participants,  particularly 
those  from  Air  Combat  Group  (ACG),  with  a  broad  range  of  training  experiences.  The 
development  and  execution  management  of  thespecificvignettes(i.e.,  portions  of  the  exercise 
scenario)  was  contracted  to  M  i  I  ski  I  Integrated  Defence  Solutions,  a  specialist  Defence  training 
company. 

Seven  vignettes  were  developed  for  PB08,  four  of  which  were  used  in  EBS08.  The  vignettes 
varied  in  terms  ofthetraining  objective,  ORBAT,  rules  of  engagement  (ROE),  airspace,  threats 
and  targets.  The  collective  nature  and  sequence  of  the  vignettes  embodied  an  escalation  in 
tension  between  the  opposing  forces,  and  the  progression  of  military  actions  over  the  three 
weeks  of  P  B08.  The  scenari  o  requi  red  BI  ue  Force  parti  ci  pants  to  conduct  a  vari  ety  of  mi  ssi  ons, 
i nd udi ng  off ensivecounter  ai r  (OCA),  off ensiveai r  support  (OAS),  deep  stri ke,  destruction  of 
enemy  air  defences  (DEAD)  and  joint  personnel  recovery  (JPR).  The  Red  Force  participants 
responded  by  empl  oyi  ng  defensive  counter  ai  r  (DCA )  tacti  cs.  Two  41WG  A  BM  teams  worked 
in  shifts  to  provide  tactical  C2supporttothelivemultinational  Red  Force  that  participated  in 
PB08.  OneoftheseABM  teams  also  participated  in  EBS08,commandingasyntheticRed  Force 
controlled  by  RAAF  SIMOPs  using  th eADGESIM  suiteof  tools. 

The  vignettes  chosen  for  days  oneto  four  of  EBS08  respectively  were  Vignette  A  (a  BI  ue  Force 
OCA  mission),  Vignette  B  (involving  BlueForceOCA  and  strike),  Vignette  D  (a  Blue  Force 
JPR  mission)  and  Vignette  E  (a  BlueForcedeep  strike  mission).  Vignette  B  was  repeated  on 
day  fi  ve  of  the  exerci  sefor  the  purpose  of  compari  ng  A  BM  team  performance  at  the  start  and 
end  of  the  activity.  The  Mi  Iski  I  WFMD  had  been  involved  in  the  development  of  the  PB08 
vignettes  and  was  to  play  a  key  White  Force  role  in  PB08.  This  promoted  the  consistency  of 
scenario  implementation  between  EBS08and  PB08.TheMilskil  member  had  previously  been 
employed  by  the  RAAF  as  a  fighter  pi  lot  and  was  abletoprovideexpertguidanceon  the  Blue 
Force  tactics  expected  in  PB08. 
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2.1.2  Participants 

The  EBS08  ABM  team  consisted  of  five  RAAF  participants  from  41WG  SRG:  an  Air  Battle 
Director  (ABD),  a  Weapons  Director  (WD)1  and  three  ABMs  in  direct  control  roles.  A  WD 
assistant  and  additional  ABM  are  sometimes  required  when  workload  is  high,  however,  the 
RAAF  was  unableto  provi dethesedueto  manning  constraints.  Although  theroles played  by 
ABMs  varied  to  suit  each  vignette,  team  membership  was  consistent  throughout  EBS08, 
allowing  team  members  to  develop  an  understanding  of  how  their  team  mates  worked  and 
how  best  to  support  them.  Theti  me  between  exercises  was  minimised  to  promotethetransfer 
of  training  benefits  from  EBS08  to  PB08.  EBS08  was  completed  two  weeks  prior  to  the 
commencement  of  PB08,  allowing  time  for  the  conduct  of  other  important  preparation 
activities. 

I  n  vi  rtual  exerci  ses  of  thi  s  ki  nd ,  the  experi  ence  I  evel  s  of  W  hi  te  Force  rol  e  pi  ayers  and  the  rati  o 
of  role  players  to  active  entities  are  significant  determinants  of  training  fidelity  (Best,  et  al„ 
2007).  During  EBS08,  theWhiteForce consisted  offiveRAAF  SIMOPsfrom41WG  SRG,  two 
assigned  to  Blue  roles  and  three  assigned  to  Red  roles,  a  Red  SIMOP  Coordinator  (RSC), 
played  by  a  DSTO  staff  member  with  extensive  ABM  experience,  and  a  WFM  D,  played  by  a 
M  i  I  ski  I  staff  member  with  operational  experi  ence  as  a  RAAF  fighter  pi  lot  and  who  had  been 
involved  in  the  development  of  the  PB08  vignettes.  The  EBS08  scenario  was  complex,  with 
SIMOPs  controlling  between  40  and  60  Red  and  Blue  aircraft.  Three  of  the  SIMOPs  had 
between  two  and  fi  ve years  experi  ence  and  two  SI  M  0  Ps  were  not  yet  qual  ifi  ed  and  were  not 
experienced  in  the  control  of  large  forces.  Although  fewer  SIMOPs  were  involved  than  was 
optimal,  no  more  personnel,  either  trained  or  in  training,  were  available.  As  a  result,  the 
workload  of  SIMOPs  was  expected  to  be  high. 

2.1.3  Exercise  staff  and  preparation 

FI  uman  factors  scientists  designed  data  col  lection  protocolsand  coordinated  theplanning  of 
EBS08.  Scenario  design  and  participant  liaison  was  supported  by  a  RAAF  ABM  instructor 
who  was  posted  to  DSTO  in  the  months  preceding  EBS08.  The  physical  components  of  the 
synthetic  training  environment  were  constructed  by  engineers  and  technicians  engaged  in 
human-in-the-loop  simulation  research.  The  software  components  were  configured  and 
supported  by  YTEK  Pty.  Ltd.  (Fishermans  Bend,  Vic)  engineers,  contracted  by  DSTO  for  the 
development  of  A  DGESIM  .  All  exercise  staff  worked  within  the  Crew  Environments  and 
Training  branch,  Air  Operations  Division,  based  at  DSTO,  Melbourne. 

RAAF  participation  in  the  exerci  sew  as  confirmed  three  weeks  prior  to  the  exercise,  and  the 
assembly  of  the  physi  cal  envi  ronment  compl  eted  two  weeks  pri  or.  The  remai  ni  ng  two  weeks 
were  spent  confi  guri  ng  and  testi  ng  the  i  nformati  on  technol  ogy  i  nfrastructure,  testi  ng  recent 
modifications  and  software  builds,  and  developing  fixes  as  problems  arose. 


1  At  the  time  the  exercise  was  conducted,  the  title  of  this  position  was  Weapons  Director.  This  title  will 
therefore  be  used  throughout  this  report.  Importantly,  the  title  of  the  position  has  since  been  changed  to 
Tactical  Director. 
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2.1.4  Exercise  schedule 

The  planned  schedule  for  EBS08  is  depicted  in  Figure  1.  At  the  beginning  of  the  week, 
parti  ci  pants  were  bri  efed  on  the  goal  sand  purposes  of  the  exerci  se,  the  exerci  se  scenari  o,  the 
structure  of  the  synthetic  training  environment  and  the  schedule  of  events.  The  ABM  team 
was  briefed  on  how  their  task  performance  and  team  processes  would  be  measured  and 
discussed  after  each  vignette.  An  initial  measurement  session  was  conducted  to  record 
participant  demographics,  experience  levels  and  their  expectations  of  the  exercise.  All 
participants  were  afforded  time  to  familiarise  themselves  with  specific  systems.  Each 
subsequent  day  of  the  exercise  consisted  of  thefollowing  key  events  in  order  of  execution: 

1.  A  scenario  update,  including  information  on  expected  threats,  ORBAT,  ROE,  airspace, 
and  enemy  intent  related  to  the  forthcoming  vignette. 

2.  Preparation  time,  in  which  the  ABM  team  could  revise  their  plans  and  the  White  Force 
team  could  discuss  the  behaviours  to  be  produced. 

3.  Mission  execution. 

4.  A  measurement  session,  requiring  ABM  team  members  and  SI  MO  Ps  to  complete  a  short 
workload  questionnaire.  In  addition,  ABM  team  members  completed  questionnaires 
addressing  team  efficacy,  team  cohesion  and  team  processes. 

5.  An  after  action  review  (AAR)  session,  to  discuss  ABM  team  task  and  teamwork 
performance,  supported  by  scenario  replay. 

6.  An  exercise  feedback  session,  where  all  RAAF  participants  took  part  in  discussions 
relating  to  the  specific  tools  and  techniques  under  evaluation  at  EBS08,  and  provided 
exercise  general  feedback.  On  Day  4,  data  was  also  collected  to  determine  the  extent  to 
which  parti  ci  pant's  expectations  of  the  exercise  were  met. 
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Figure  1:  The  schedule  of  events  for  days  lto5  ofEBS08 
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2.14.1  Unscheduled  events 

Mission  1  was  disrupted  and  Mission  2  was  partially  degraded  due  to  instability  in  the 
simulation  software.  In  order  to  reduce  the  risk  to  the  remaining  vignettes  and  allow  further 
software  testing,  an  improvised  'air  warfare  game'  was  conducted  in  placeof  theAAR  session 
on  Day  2.  This  game  requi red  each  A  BM  to  pai  r  with  a  SI  M  OP  and  control  two  fi ghter  pai  rs. 
The  aim  of  the  game  was  to  be  the  first  formation  to  fly  through  the  centre  of  an  arbitrary 
hexagonal  airspace  and  back  to  their  starting  corner,  without  being  destroyed  by  an  opposing 
formation.  The  participants  reported  that  this  session  was  a  useful  alternative,  as  it  allowed 
the  ABMs  to  practice  basic  fighter  controls  skills,  and  it  granted  the  SIMOPs  additional 
p  racti  ce  w  i  th  th  e  h  u  man-  mach  i  n  e  i  nterf  ace  of  th  e  I  atest  si  mu  I  ati  on  tool  s.  Th  e  softw  are  p  roved 
to  be  stable  throughout  the  game  and  the  remaining  vignettes. 

2.1.5  Procedure 

DSTO  human  factors  scientists  were  responsible  for  managing:  adherence  to  the  exercise 
schedule,  thedelivery  of  briefings  and  associated  materials,  and  access  to  the  facility  by  staff, 
participants  and  visitors.  Thefollowing  paragraphs  descri  be  other  exerciseroles  performed  by 
parti  ci  pants  and  the  exerci  se  management  team. 

2.15.1  Participant  roles  and  responsibilities 

The  ABM  team  was  required  to  command  Red  Force  air  assets  to  defend  airspace  and  key 
poi  nts  agai  nst  the  I  arger  and  techni  cal  ly  superi  or  Bl  ue  Force.  A  Ithough  the  A  BM  team  was  not 
expected  to  prevent  the  I  oss  of  al  I  key  poi  nts  or  assets,  the  effecti  veness  of  thei  r  defence  was 
assessed  by  the  RAAF  ABM  instructor  (from  a  task  work  perspective)  and  theABD  (from  a 
teamwork  perspective).  Both  assessors  monitored  the  same  situation  display  and 
communication  channels  available  to  the  ABM  team,  and  utilised  the  assessment  tools 
described  later  in  this  report.  In  an  effort  to  promote  the  consistency  of  measurement  across 
EBS08  and  PB08,  the  same  ABM  instructor  was  tasked  with  assessing  ABM  team  task 
performance  at  the  two  exerci  ses. 

The  Red  SIMOPs  were  responsible  for  manipulating  the  synthetic  fighter  assets  under  the 
control  oftheABM  team,  and  emulating  the  communications  of  those  aircraft  pilots.  Dueto 
thelargepool  of  simulated  aircraft  to  be  control  led  (generally  between  four  and  eight  aircraft 
per  SI  MOP),  a  DSTO  staff  member,  formerly  employed  as  a  RAAF  ABM,  fulfil  led  the  role  of 
RSC.  This  involved  'scrambling'  the  fighter  formations  requested  by  the  ABM  team,  and 
hand  i  ng  them  over  to  the  SI  M  0  P  w  hose  apparent  w  orkl  oad  w  as  I  owest  at  that  ti  me.  Si  mi  I  arl  y , 
the  RSC  would  take  control  of  aircraft  which  were  'kill -removed',  returning  to  base  or 
requ  i  ri  ng  ai  r-to-ai  r  refuel  I  i  ng.  Thi  s  permitted  SI  M  O  Ps  to  focus  thei  r  efforts  on  mi  ssi  on  phases 
requiring  their  domain  knowledge  and  experience,  rather  than  benign  phases. 

The  Bl  ueSI  M  OPs  were  requi  red  to  mani  pul  atetheBI  ue  Force  ai  r  assets  under  thedi  recti  on  of 
the  WFM  D.  Due  to  the  large  number  of  assets  (generally  between  12  and  20  aircraft  per 
SI  M  OP),  theSIM  OPs  were  requi  red  to  mani  pul  ate  formations  as  chained  groups,  that  is,  with 
the  lead  group  being  used  to  control  all  trailing  groups.  In  terms  of  workload,  the 
management  of  large  numbers  of  aircraft  was  only  possible  because  these  SI  M  OPs  were  not 
required  to  communicate  with  external  parties.  The  WFM  D  directed  the  Blue  SIMOPs  to 
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arrange  different  aircraft  types  (with  associated  roles)  into  a  certain  order  and  position  them  at 
a  specific  location  and  time.  In  this  way  the  WFMD  was  able  to  create  a  scenario 
representative  of  the  Blue  Force  tactics  expected  in  PB08. 

2.15.2  AAR  session 

After  each  vignette,  the  ABM  instructor  and  ABD  debriefed  the  ABM  team  on  their  task  work 
and  teamwork  performance  respectively,  facilitating  discussions  of  key  strengths  and 
weaknesses.  The  task  assessor  replayed  relevant  portions  of  the  mission  to  support  ratings 
and  comments  against  performance  measures.  These  replays  represented  scenario  ground 
truth,  that  i  s,  acti  vity  in  thefl  i  ght  envi  ronment  as  it  actual  I  y  occurred,  and  user-sped  fi  c  vi  ews, 
that  is,  activity  in  theflight  environment  as  it  appeared  on  a  specific  user's  screen.  Ground 
truth  differed  from  user-specific  views  as  a  result  of  quality  and  range  of  thesimulated  radar. 
Participant  communications  were  also  recorded  during  execution  and  could  be  replayed  in 
synchrony  with  video  replays.  The  ABD  then  discussed  the  quality  of  team  processes.  At  all 
stages  the  evaluations  were  open  to  comment  and  discussion  by  all  ABM  team  members. 
PI  ease  refer  to  Subsection  2.3.5for  a  more  detailed  discussion  oftheapproach  employed  and 
tools  utilised  during  theAAR. 

2.15.3  Data  collection 

FI  uman  factors  scientists  col  I  ected  data  regardi  ng  the  suitabi  I  ity  of  thetrai  ni  ng  envi  ronment 
and  the  value  of  the  exercise.  Qualitative  and  quantitative  data  was  entered  by  participants 
i  nto  el  ectroni  c  forms  at  the  begi  nni  ng  of  the  exerci  se  and  after  every  mi  ssi  on.  Thi  s  i  nformati  on 
included  participant  demographics,  qualifications  and  experience,  exercise  expectations,  post¬ 
exercise  evaluations  of  whether  expectations  were  met,  workload,  team  efficacy  and  team 
cohesion.  This  data,  along  with  data  obtained  fromtheteamwork  and  taskwork  assessments, 
will  contribute  to  discussions  on  the  effectiveness  of  EBS08  in  preparing  the  ABM  team  for 
PB08 (Shanahan,  etal.,  in  preparation). 

Data  was  also  collected  to  assist  in  the  evaluations  of  a  number  of  tools  and  techniques, 
namelyth  eADGESIM  suiteof  tools,  the!  oteboard,  the  A 1/1/  AR  tool,th  eTDT  framework  and 
theAAR  tools.  These  tools  and  techniques  will  be  described  in  detail  in  Subsection  2.3. 
Qualitative  data  was  obtained  through  semi-structured  interviews  with  RAAF  personnel 
regardi  ng  the  suitabi  I  ity  of  thetool  s  and  techni  ques  for  use  i  n  a  RA  A  F  trai  ni  ng  envi  ronment. 
Information  obtained  through  these  interviews  forms  the  basis  for  discussion  in  Section  3  of 
this  report. 

2.2  Exercise  infrastructure 

The  fol  I  ow  i  ng  su  bsecti  ons  provi  de  a  descri  pti  on  of  the  I  ayout  of  the  exerci  se  faci  I  i  ty  and  the 
components  of  the  information  technology  infrastructure. 

2.2.1  General  facility  layout 

The  general  I  ayout  of  the  si  mu  I  ati  on  faci  I  ity  is  depicted  in  Figure2.  Asshown,thefacility  was 
split  into  four  areas:  theserver  room,  the  White  Forcecontrol  centre,  thetacti  cal  control  centre 
and  the  observation  room.  The  server  room  was  separated  from  the  other  components  by 
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solid  walls,  whilethe  other  areas  were  separated  by  two  metre  high  partitions.  The  server 
room  contained  th eADGESIM  simulation  and  file  servers.  TheSIMOPs  and  WFMD  were 
situated  in  the  White  Force  control  centre,  which  was  arranged  by  Force.  The  ABM  s,  WD, 
ABD  and  RAAF  instructor  weresituated  inthetactical  control  centre.  TheWD  was  positioned 
such  that  he  could  monitor  the  three  subordinate  ABMs.  The  observation  room  was 
constructed  to  allow  the  exercise  management  team  and  visitors  to  view  activity  insidethe 
tactical  control  centre  unobtrusively  via  live  feed  projections  and  a  plasma  display. 


Figure2:  A  top-down  representation  of  the  simulation  facility,  showing  the  control  cells  and 
positioning  of  personnel 

2.2.2  Information  technology  infrastructure 
2.2.2.1  Common  systems 

All  exercise  participants  were  provided  with  two  separate  personal  computers  (PCs)  on  a 
common  network:  oneasthecommunications  console,  the  other  as  the  main  workstation.  The 
communicationsconsoleconsisted  of  touch  screen  monitors,  push-to-talk  pedal  and  a  headset. 
An  ADGESIM  application  called  DI5V0X  was  operated  on  this  console.  This  provided  basic 
radio  and  intercom  simulation  and  a  graphical  user  interface  similar  to  the  operational 
systems  used  by  the  RAAF.  A  single  communications  configuration  was  developed  which 
included  a  unique  operator  role  for  each  physical  position.  The  software  utilises  the 
D  i  stri  buted  I  nteracti  ve  Si  mu  I  ati  on  ( D I S)  protocol  w  hi  ch  al  I  ow  s  parti  ci  pant  commu  ni  cati  ons  to 
be  I  ogged  w  i  th  any  D I S I  oggi  ng  system.  The  mai  n  workstati  on  consi  sted  d  ual  monitors  (each 
with  a  resolution  of  1280x1024).  The  software  being  operated  depended  on  the  participant 
role. 


7 


DSTO-TR-2305 


Figure  3:  TheWhiteF orce  con  trol  cen  tre,  show ing  the  posi ti oningofwhi te  force  person n el  and  the 
layout  of  workspaces 


2.2.22  Scenario  generation  systems 

The  White  Force  control  centre  is  represented  in  Figure  3.  All  members  of  the  White  Force 
w  ere  p  rov  i  d  ed  w  i  th  th  e  P  i  I  ot  Si  mu  I  ati  on  I  nterf  ace  (P  SI)  softw  are,  w  h  i  ch  al  I  ow  s  enti  ti  es  to  be 
created,  controlled  and  transferred  between  team  members.  Pleasereferto  Subsection  2.3.1for 
a  moredetailed  description  of  PS/.  Each  instance  of  P5I  was  connected  to  a  server  operating 
the  VR-Forces  software,  which  models  all  of  the  aircraft  in  the  si  mu  I  ati  on  environment  and 
broadcaststhisviatheDIS  protocol.  TheWFMD  also  used  afreevideo  network  cl  ient(VNC), 
called  U  ItraVN  C,  to  monitor  thesituati on  display  used  by  the WD.  This  provided  the WFMD 
awareness  of  when  Blueforcesweredetected  and  what  rules  of  engagement  might  beapplied, 
at  any  point  in  time.  The  RSC  also  used  a  shared  instance  of  th eToteboard  application  to 
determine  when  to  create  entities  that  had  been  scrambled  by  the  ABM  team. 

2.2.2.3  Command  and  control  systems 

All  members  of  the  ABM  team  and  the  RAAF  instructor  assessing  task  work  performance 
were  provided  with  the  Solipsys  tactical  situation  display  software.  The  software  was 
configured  with  airspace  and  waypoint  data,  and  a  RAAF-developed  user  profile  which 
d  ef  i  n  es  asp  ects  su  ch  as  d  ata  bl  ock  f  i  el  d  s,  i  d  enti  fy  f  ri  en  d  or  f  oe  ( I F  F )  d  ecod  i  ng  an  d  a  mu  I  ti  tu  d  e 
of  appearance  settings.  Each  instance  of  Solipsys  was  connected  to  the  multiple  source 
correl  ator/  tracker  (M  SCT),  w  hi  ch  i  n  turn  was  sti  mul  ated  by  an  ADGESIM  server  appl  i  cati  on 
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called  SensorLink.  SensorLink  was  used  to  model  the  PB08  radar  types  and  sites,  and  to 
translate  the  detection  of  entities  received  via  the  DIS  protocol  to  a  plot  protocol  compatible 
with  the  M  SCT.  This  accurately  repl  i  cates  the  operational  C2  system  used  by  the  RAAF,  with 
the  exception  that  SensorLink  replaces  the  real  sensor  network.  Each  member  was  also 
provided  with  a  shared  instance  of  the  7“ otaboard  application.  This  was  used  to  collect  and 
disseminatestatus  information  concerning  aircraft  assigned  to  team  members.  During  mission 
execution,  projections  land  2  in  Figure4  replicated  theSolipsysand  Toteboard  displays  of  the 
WD. 


FC3Workstation 


FC1  Workstation 


SMART  Board 


Projection  1 


Projection  2 


WD  Workstation 


Figure4:  The  tactical  control  centre,  showing  the  positioning  of  members  of  the  ABM  team,  the 
layout  of  operator  workstations  and  the  visual  tools  used  during  AAR 


2.2.24  AAR  infrastructure 

The  AAR  infrastructure  consisted  of  two  parts:  thecol  lection  system,  which  was  used  during 
mission  execution,  and  the  review  system,  which  was  used  during  the  AAR.  Both  systems 
utilised  components  of  theADGESIM  Tactical  AfterAction  Review  for  Distributed  Interactive 
Simulation  ( TAARDIS )  suite.  The  collection  system  was  composed  of  a  DIS  logging  and 
bookmarking  application  that  operated  on  the  task  assessor's  workstation,  and  individual 
screen  capture  applications  that  operated  on  each  ABM  team  member  workstation.  Thetask 
assessor  was  also  provided  with  the4  1/1/4  R  tool  for  prompted  performance  measurement  and 
the  Toteboard  application  for  situation  awareness  (SA)  during  mission  execution.  Due  to  the 
number  and  nature  of  these  appl  icati  ons  and  the  requi  rement  for  concurrent  access,  thetask 
assessor's  workstation  was  fitted  with  four  monitors.  The  review  system  included  a  rear 
projecti  on  SM  A  RT  board  (see  Fi  gure  4)  to  di  spl  ay  the  rati  ngs  and  comments  made  by  thetask 
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assessor  using  the  A  WAR  tool.  The  task  assessor's  comments  were  supported  by  wall 
projections  and  audio  replays  of  relevant  mission  segments  that  were  control  led  by  an  AAR 
operator  from  thetask  assessor's  workstation. 


Projection  3 


Projection  4 


Plasma  Display 


Listening  Post 


Self  Briefing  Area 


Figure5:  The  observation  room,  showi  ng  thetools  avai  lableto  observers  to  aid  in  their  understanding 
of  participant  activity 


2.2.3  Observation  room 

The  observation  room  was  constructed  to  providetheexercisemanagementteam  and  visitors 
with  an  area  from  which  they  could  view  the  exercise  without  disturbing  the  RAAF 
participants.  Figure  5  shows  the  arrangement  of  the  observation  room.  Observers  were 
provi  ded  with  a  number  of  tool  s  to  assist  i  n  thei  r  understandi  ng  of  missi  on  events  and  team 
i  nteracti  ons  duri  ng  the  mi  ssi  ons  and  A  A  R.  A  s  observers  were  most  i  nterested  i  n  the  activity 
within  the  ABM  team,  a  real-time  image  of  the  team  was  displayed  on  the  plasma  screen 
located  within  the  observation  room.  The  input  to  this  display  was  from  oneof  two  cameras, 
Camera  A  and  Camera  B,  located  within  the  tactical  control  centre  (refer  to  Figure  2).  In 
Figure  5,  the  i  nputto  the  plasma  display  isfrom  Camera  A.  Input  to  thedisplay  was  modified 
to  provide  the  best  view  of  activity  within  the  tactical  control  centre.  In  general,  Camera  A 
provided  the  best  view  during  missions  and  Camera  B  provided  the  best  view  of  AARs 
because  it  showed  thetask  assessor  referring  to  various  visual  aids  on  the  projected  images 
and  SMART  Board. 

Two  images  were  projected  ontothewall  in  the  observation  room.  Projection  3  (see  Figure  5) 
wastheWD  tactical  situation  display,  which  was  updated  in  real  time  and  showed  thefocus 
of  the  WD's  attention.  Projection  4showed  ground  truth  data  and  was  also  updated  in  real 
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time.  A  listening  post  was  available,  providing  five  headphones  for  observers  to  listen  to 
communication  within  and  between  thetactical  control  centreand  White Forcecontrol  centre. 
Observers  could  select  the  communications  channels  of  interest  using  the  communications 
system  interface  on  thecomputer  provided  i  n  the  self  briefi  ng  area.  This  computer  could  also 
display  ground  truth  and  could  be  used  to  zoom  or  pan  to  areas  of  interest  on  the  ground 
truth  display. 

2.3  Tools  and  techniques 

A  major  goal  of  EBS08  was  to  provide  an  environment  within  which  a  range  of  tools  and 
techniques  could  be  evaluated  and  developed  for  future  usein  RAAF  training  exercises.  The 
following  paragraphs  descri  be  thetoolsthatwereunder  evaluation  in  EBS08  in  terms  of  their 
use  and  the  improvements  they  may  offer  over  current  RAAF  operations. 

2.3.1  Th eADGESIM  simulation  suite 

Th  eADGESIM  simulation  system  is  comprised  oftheD  ISVOX  radio  simulator  (described  in 
Subsection  2.2.1),  PSI ,  theSensorLIN K  simulated  sensor  gateway  (described  in  Subsection 
2. 2.2. 3),  the  Si  mM  an  si  mu  I  ator  manager  and  theT acti  cal  After  A  cti  on  Revi  ew  for  D  i  stri  buted 
Interactive  Simulation  ( TAARDIS ;  described  in  Subsection  2.3.5).  The  nature  of  the  PSI 
softw  areinfluencedSIMOP  beh  av  i  ou  r  an  d  con  seq  u  entl  y  th  e  f  i  d  el  i  ty  of  th  e  trai  n  i  n  g  p  rov  i  d  ed 
to  the  ABMs.  For  this  reason,  the  properties  of  PSI  are  explained  in  further  detail  in 
Subsections  2.3.11  and  2.3.12. 

2.3.11  Th ePSI  user  interface 

The  PSI  user  interface  is  represented  in  Figure  6.  Within  PSI,  the  simulation  scenario  is 
displayed  on  a  3D  world  map  with  all  objects  drawn  at  ground  level.  Map  overlays  depicting 
thecoastline,  airspace,  waypoints,  routes  and  ground  based  radar  coverage  may  be  selectively 
displayed.  Entities  are  drawn  as  small  circles  with  history  trails  and  velocity  leaders,  all  of 
which  may  be  col  our  coded  to  represent  the  force,  domain  or  identity  of  the  entity.  Absolute 
and  relative  cursor  position  is  displayed,  and  a  bearing  and  range  line  tool  is  provided  for 
measuring  the  relative  position  of,  and  distance  between,  entities.  Alpha-numeric  data  block 
fields  are  drawn  adjacent  to  entities,  describing  actual  or  derived  state  parameters.  These 
fields  are  categorised  as  identity,  position,  targeting  or  missile  related.  A  table  summarising 
the  IFF  modes  and  codes,  weapon  and  fuel  state,  velocity,  altitudeand  current  task  of  entities 
assigned  to  each  instance  of  PSI  is  also  displayed  at  the  top  of  the  panel. 

Sensor  coverage  is  depicted  asoneor  moretransparent  zones  around  each  entity  on  the  world 
map,  coloured  by  sensor  type(as  shown  in  Figure6).  Each  entity  has  a  visual  sensor  which  is 
always  on  and,  if  fitted,  radar  and/  or  infrared  sensors  which  may  be  exclusively  enabled. 
Entiti  es  withi  n  a  coveragezoneof  another  entity  areconsidered  to  have  been  detected  by  that 
entity.  The  supported  types  and  modes  of  each  sensor  as  well  as  the  associated  range,  angleof 
azimuth  and  elevation  constraints  may  be  set  for  each  entity  type.  For  entities  in  formation, 
the  sensor  type  in  use  can  be  synchronised  across  formation  members,  and  thecoveragezones 
aredrawn  as  a  merged  volume.  Alternatively  thecoveragezones  of  distinct  formations  are 
drawn  as  overlapping  volumes.  The  state  of  'weapon  locks'  between  entities,  in  theform  of 
pai  ri  ng  I  i  nes,  warni  ngs,  weapon  sel  ecti  on,  weapon  empl  oyment  range  and  i  mmi  nent  I  oss  of 
'lock'  indications,  are  also  displayed  on  the  map. 
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!  Pilot  Interface  Client  3.3.7  (  185:131:105  ) 
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Figure6:  The  PSI  user  intaTace  dqoicting  a  formation  of  friendly  entities  i  nta'cgjti  ng  an  energy  formation.  Shaded  areas  shew  sensor  coverage 
approximations  and  weapons  aigaganmt  zones. 
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PSI  provides  several  mechanisms  by  which  SI  MOPs  may  control  entities.  A  user  may  quickly 
switch  control  between  entiti  es  by  sel  ecti  ng  an  entity  on  the  map  or  cyd  i  ng  through  theentity 
list  with  key  commands.  A  SI  MOP  may  manipulate  the  speed,  altitude,  and  heading  of  an 
enti  ty  or  formati  on  vi  a  si  mp I  e  poi  nt  and  cl  i  ck  control  s  on  or  adj acent  to  the  map .  The  n u mber 
of  cl  i  cks  determi  nes  the  rate  of  change  i  n  the  d  i  mensi  on  specifi  ed .  For  i  nstance,  a  si  ngl  e  cl  i  ck 
speed  change  uses  the 'military  power'  throttle  condition  and  a  doubleclick  uses  the 'after 
burner'  throttlecondition.Themagnitudeof  the  command  relativetothecurrent  setting  may 
influence  the  manner  in  which  theentity  attains  the  desired  state.  For  example,  a  formati  on 
will  perform  an  immediate  'in  place'  manoeuvre,  for  a  -180  degree  turn  or  a  'delay' 
manoeuvre  for  a  ~00  degree  turn.  Entities  may  be  tasked  to  fly  a  holding  pattern,  or  follow  a 
3D  route  or  another  entity.  Formations  may  be  synchronised  in  terms  of  their  sensor  modes, 
weapon  types  and  time  of  engagement.  An  entity  may  target  another  entity  within  its  sensor 
coverage  by  right  clicking  on  the  target,  and  observe  the  state  of  weapon  acquisition  via 
distinct  missileentity  symbology. 

A  typical  pattern  of  control  in  a  simulation  session  would  involve  the  following  phases. 
Firstly,  a  SI  MOP  would  configurethePS/  display,  in  terms  of  overlays,  map  range  and  centre, 
data  block  fields  and  entity  colour  coding.  Then  a  SI  MOP  would  create  enti  ties  required  for 
the  scenario  from  a  script  defining  the  starting  states  and  tasks,  or  assign  entities  that  had 
already  been  created.  Subsequently,  the  SI  MOP  is  able  to  select  an  entity  or  formation  and 
modify  thei  r  fl  i  ght  parameters,  weapons  system  state  or  unattended  tasks  as  descri  bed  above 
and  as  required  by  the  scenario.  Finally  a  SIMOP  may  either  destroy  or  release  entities  to 
other  SI  MOPs  if  no  longer  required  for  their  direct  control. 

2.3.12  PSI  enhancements  for  EBS08 

Each  entity  is  accompanied  by  a  data  block  listing  its  various  characteristics  and  parameters. 
PSI  considers  ownership  and  context  when  selecting  information  to  display  in  each  data 
block.  Whilst  all  entities  are  created  on  a  central  VR-Forces  server,  they  may  only  b  eassigned 
to  (that  i  s,  ow  ned  by)  a  si  ngl  e  SI  M  O  P  at  a  ti  me.  I  nformati  on  concerni  ng  the  sensors,  weapons 
and  fuel  state  of  an  entity  areknown  only  to  theowner  of  that  entity.  Therelativepositi  on  and 
aspect  of  two  entities,  which  may  be  required  for  tasks  such  as  weapon  targeting,  may  not  be 
known  uni  ess  one  entity  i  s  attached  (that  i  s,  the  SI  M  OP  control  I  i  ng  that  entity  has  selected  that 
ai  rcraft  as  the  frame  of  reference).  Information  is  also  filtered  based  on  context.  Thefirst  level  of 
context  rel  ates  to  w  h  ether  entity  per  speed  ve  i  s  appl  i  ed ,  that  i  s,  w  h  ether  al  I  of  the  entiti  es  i  n  a 
scenario  are  displayed  or  just  those  that  are  assigned  or  detected  by  assigned  entities.  The 
second  level  relates  to  the  type  of  sensor  volume  an  entity  is  located  within.  For  example, 
some  positi  on  fi  el  ds  sw  itch  on  and  off  as  the  range  between  entiti  es  i  ncreases  or  decreases  (to 
display  the  units  and  frame  of  reference  required  by  doctrine),  and  some  identity  fields  are 
disabled  becausea  given  sensor  type  is  not  capable  of  determining  a  given  parameter. 

The  context-based  rules  used  to  determine  which  information  is  displayed  in  data  blocks  in 
the  I  atest  versi  on  of  ADGESIM  are  consi  derabl  y  d  ifferent  to  those  i  mpl  emented  i  n  the  versi  on 
of  PSI  employed  atSACTU  and  3Controlling  and  Reporting  Unit(3CRU).  The  aim  was  to 
i  mp  rove  the  consi  stency  of  SI  M  0  P  behavi  our  by  restri  cti  ng  avai  I  abl  e  i  nformati  on  to  only  that 
which  could  reasonably  be  known  by  a  pilot  in  that  context.  This  change  was  expected  to 
reduce  SIMOP  workload.  Whilst  the  modifications  achieved  this  aim,  the  difference  in 
information  requirements  of  SI  MOPs  and  pilots  was  not  considered.  The  EBS08  SI  MOPs 
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would  have  benefited  from  a  higher  level  of  SA  to  counteract  the  additional  responsibility  of 
coordinating  the  execution  of  the  training  scenario  with  other  White  Force  members,  a 
responsi  bil  ity  that  woul  d  not  be  requi  red  of  the  pi  I  ots  the  SI  M  0  Ps  represented .  The  overal  I 
effect  of  th  e  mod  i  f  i  cati  on  to  th  e  P  SI  softw  are  w  as  to  create  ahigherlevelofSIMOPworkload 
than  would  have  been  experienced  with  the  previous  version  of  ADGESIM  . 

2.3.2  T  oteboard 

TheToteboard  (example  screenshot  shown  in  Figure  7)  is  an  electronic  tool  used  to  providethe 
ABM  team  with  information  on  mission  parameters,  such  as  aircraft  fuel  states,  alert  states, 
weapon  loadouts,  assigned  airspace  and  radio  frequency,  tanker  offload  levels,  endurance 
estimates,  and  aircraft  turnaround  timings.  TheToteboard  is  a  customised  Microsoft  Excel 
spread  sheet  that  performs  embedded  cal  culations  using  Visual  Basicfor  A  ppli  cations  {VBA ) 
scripting.  It  was  originally  developed  atSACTU  for  use  in  training  events  conducted  in  the 
SACTU  simulator,  but  has  since  been  utilised  by  the  RAAF  in  a  number  of  live  military 
exerci  ses.  TheT oteboard  i  s  a  shared  spreadsheet  that  al  I  ows  every  member  of  an  A  BM  team  to 
update  information.  As  responsibilities  are  divided  across  members  of  an  ABM  team,  this 
permits  information  on  specific  formations  to  be  updated  by  the  team  member  controlling 
those  formations,  circumventing  doublehandling  and  reducing  the  risk  of  error. 

2.3.3  The TDT  framework 

TDT  was  developed  by  the  US  Navy  in  an  effort  to  improve  teamwork  processes  (Smith- 
Jentsch,Zeisig,  Acton  &  McPherson,  1998).Thegoal  ofTDT  is  to  improveteam  performance 
in  four  areas:  information  exchange,  communication,  supporting  behaviour  and 
initiative  leadership.  Table  1  provides  examples  of  good  performance  under  each  of  these 
categories.  The  teamwork  categories  provide  a  framework  for  the  structure  of  the  TDT 
process. 


T  able  1:  The  four  areas  of  teamwork  considered  in  theTDT  framework 


Information  exchange: 

-  Utilising  all  sources  of  information, 

-  Passing  information  without  being  asked, 

-  Providing  big  picture  situation  updates. 

Communication  delivery: 

-  Using  correct  phraseology, 

-  Providing  complete  reports, 

-  Avoiding  excess  chatter, 

-  Communicating  all  information  clearly. 

Supporting  behaviour: 

-  Monitoring  and  correcting  team  errors, 

-  Providing  and  requesting  backup  or  assistance 
to  balanceworkload. 

Initiative/leadership: 

-  Providing  guidance  or  suggestions  to  team 
mates, 

-  Stating  clear  and  appropriate  priorities. 

TheTDT  proced u re  w as  not  f u 1 1 y  i  mp I emented  i  n  E BS08,  w i th  the categori es  servi  ng  onl y  to 
provide  a  common  vocabulary  for  assessment,  feedback  and  discussion  of  team  processes 
withi  n  thecontext  of  the  exercise.  The  A  BM  team  was  i  ntroduced  to  theTD  T  categories  at  the 
beginning  of  the  exercise.  They  were  provided  with  a  written  summary  of  an  accident 
involving  two  US  Army  Blackhawk  helicopters  being  shot  down  (included  in  Appendix  D) 
and  were  asked  to  discuss  the  incident,  identifying  positive  and  negative  examples  of 
behaviours  related  to  the  four  TDT  teamwork  categories.  The  ABD  rated  the  teamwork 
p  erf  or  man  ce  of  th  e  A  B  M  team  u  n  d  er  each  of  th  e  categori  es  d  u  r  i  n  g  each  mi  ssi  on ,  p  rov  i  d  i  n  g  a 
teamwork  debrief  to  the  team  after  the  completion  of  each  mission. 
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Figure7:  TheToteboard  used  in  EB508.  Thercwsin  thespreadshedt  correspond  to  sped  fi  c  entities  and  thecd  umns  to  ther  correspond/'  ng  paramate's. 
N ote  that  thecdour-coding  cf  entities  by  callsign  matches  thecdour  theseentities  will  appear  in  P5I. 
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2.3.4  TheA  WAR  tool 

TheA 1/1/4  R  tool  provides  a  means  by  which  to  capture  team  performance  evaluations  and 
provide  structured  debriefs.  Itwas  created  under  AIR  07/ 232  by  customising  a  set  of  Excel 
spreadsheets  usi  ng  i/ BA  scri  pti  ng.  D uri  ng  past  exerci ses  i  n  thi  s  research  program,  the  M  entor 
tool,  developed  by  Calytrix  Technologies  (Fyshwick,  ACT),  has  been  used  for  this  purpose 
(e.g„  Best  &  Burchat,  2006;  Best,  et  al„  2007).  However,  a  tailored  tool  was  required  on  this 
occasion  to  meetthespecific  requirements  of  EBS08.TheA 1/1/4  R  tool  can  be  populated  with  an 
extensive  list  of  fine  measures  that  can  be  used  repeatedly  across  trials,  allowing  greater 
breadth,  depth  and  consistency  of  ratings.  The  categories  and  measures  also  serve  as  focus 
areas  around  which  an  assessor  may  structure  task  performance  debriefs. 

For  EBS08,the4 1/1/4  R  tool  was  populated  with  theABM  DCA  team  goal  hierarchy,  which  was 
derived  from  the  Australian  Joint  Essential  Tasks  (ASJETs).  TheASJETs  (McCarthy  et  al., 
2003)  are  a  comprehensive  list  of  all  tasks  essential  to  the  preparation,  planning  and  conduct 
of  Australian  Defence  Force  (ADF)  operations.  The  ABM  DCA  team  goal  hierarchy 
(Hasenbosch  &  Best,  2007)  is  an  extrapolation  of  a  subset  of  the  ASJETs,  including  only  those 
goal  s  that  rel  ate  to  the  work  of  A  BM  teams  cond  ucti  ng  D  C  A  mi  ssi  ons.  The  hi  erarchy  consi  sts 
of  six  team  goal  categories  as  well  as  subordinate  goals,  subgoals,  and  criteria.  The 
relationship  between  levels  of  the  hierarchy  is  such  that  the  lower  level  goals  and  subgoals 
explain  how  the  higher  level  goal  sand  goal  categories  are  achieved  and  the  higher  level  goals 
and  goal  categories  explain  why  the  lower  level  goals  and  goal  categories  are  important  to 
achieve.  The  criteria  provide  a  description  of  what  constitutes  good  performance  on  each 
subgoal.  The  hierarchy  represents  the  work  of  theABM  teamin  DCA  missions  in  away  that 
facilitates  understanding  and  reporting  of  how  that  work  contributes  to  the  higher-level 
organisational  goals  of  the  ADF. 

2.3. 4.1  Performance  assessment 

The  training  assessment  approach  currently  employed  in  the  RAAF  involves  subject  matter 
experts  (SM  Es)  rati  ng  trai  nee  performance  as  competent  (C )  or  not  yet  competent  ( N  YC )  on  a 
small  number  of  broad  categories,  with  extensive  notes  detailing  the  positive  and  negative 
aspects  of  performance.  Guideli  nes  areprovided  for  each  category  to  i  ncrease  the  consistency 
of  rati  ngs,  however,  rati  ngs  sti  1 1  rely  heavi  ly  on  the  SM  E's  experi  enceand  may  vary  between 
assessors  dueto  different  preferences  and  different  experiences  of  what  works  and  doesn't 
work.  There  i  s  no  requi  rement  to  rateal  I  measures,  and  assessors  are  free  to  focus  on  the  areas 
they  see  as  most  i  important.  Dueto  thisfl  exi  bil  ity,  rati  ngs  may  even  vary  for  thesame  assessor 
over  time.  As  a  result,  it  can  be  difficult  to  make  valid  performance  comparisons  between 
students  of  a  si  ngl  e  course  or  to  detect  the  performance  i  improvement  of  a  student  or  cohort 
over  ti  me.  A I  though  the  operators  agree  that  the  consi  stency  and  objecti  vi  ty  of  rati  ngs  shou  I  d 
be  improved,  they  value  the  flexibility  of  the  current  system  and  the  avenue  it  provides  for 
experts  to  pass  on  thei  r  experi  ences  to  students. 

A  screenshot  of  the4  WAR  tool,  populated  with  theABM  DCA  team  goal  hierarchy,  is  shown 
in  Figure  8.  As  shown,  thesixteam  goal  categories  can  be  expanded  to  reveal  the  constituent 
team  goals  and  team  sub-goals.  Each  sub-goal  is  accompanied  by  a  criterion  for  rating  team 
performance.  Assessors  rate  performance  by  selecting  a  score  from  a  drop  down  menu. They 
can  also  provide  explanatory  remarks.  In  EBS08,  the  assessor  selected  a  score  from  the 
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fol  I  ow i  ng  f i  ve  opti  ons:  1  =T erri  bl  e,  2  =  Poor,  3  =  A dequate,  4  =  Good ,  and  5  =  Excel  I  ent.  Su fa- 
goal  s  th  at  w  ere  n  ot  rated  retai  n  ed  th  e  d  ef  au  1 1  score  of  0.  F  i  gu  re  8  d  ep  i  cts  p  erf  or  man  ce  on  S  u  b- 
G  oal  5.1.4:  M  an  age  A  A  R  operations  bei  ng  rated  as  T  erri  ble'.  The  assessor  could  also  label  overal  I 
team  goals  as  strengths  or  weaknesses  by  depressing  the  'strength'  or  ' weakness '  button 
corresponding  to  the  relevant  goal.  Some  sub-goals  relating  to  safety,  for  exampleSuib-Goa/ 
5.2.1:  Apply  Current  ROE ,  werelabelled  as  critical,  and  a  rating  of  lor  2  would  result  in  the 
team  failing  the  overall  mission.  This  'critical  fail'  function  could  be  switched  off  if 
appropriate. 
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Figure  8:  The  AW  A  R  assessment  user  interface.  Goal  category  5  has  been  expanded  to  reveal  the 
constituent  goals  and  sub-goals.  The  figure  also  shows  thedropdown  rating  menu.  Note 
that  the  comments  included  in  this  figure  are  fictional  and  are  not  those  made  by  the 
assessor  during  EB508. 


The  DCA  team  goal  hierarchy  was  incorporated  into  theA  WAR  tool  with  the  intention  of 
i  ncreasi  ng  theconsi  stency  of  performance  assessments  and  the  specificity  of  feedback  over  the 
assessment  approach  currently  employed  by  the  RA  A  F.  Inclusion  oftheteamgoal  hierarchy 
prompted  the  assessor  to  rate  performance  on  a  widerangeof  specific  sub-goals,  rather  than 
broad  multifaceted  goals,  and  performance  criteria  provided  guidance  on  what  constituted 
good  performance  on  each  sub-goal,  promoting  the  consistency  of  performance  assessment 
across  missions.  The  same  goal  hierarchy  was  used  for  the  duration  of  EBS08and  PB08,  and 
therefore,  A  BM  teams  were  rated  on  thesame  listof  measures  on  al  I  occasi  ons.  Thi  s  permitted 
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the  comparison  of  performance  across  missions,  across  exercises  and  between  teams.  Note 
that  due  to  time  restrictions,  the  task  assessor  was  able  to  rate  performance  on  most,  but  not 
all,  sub-goals.  Use  of  the  team  goal  hierarchy  served  to  highlight  the  relationship  between 
specificteam  behaviours  and  high  level  ADF  goals,  thereby  raising  the  team's  awareness  of 
the  importance  of  each  sub-goal.  It  also  provided  aframeworkfor  providing  the  team  with 
performance  feed  back. 

23.4.2  Performance  review 

A  screenshot  of  the4  WAR  debrief  user  interface  is  shown  in  Figured  In  this  figure,  Team  Goal 
3.2:  Manage  Equipment  has  been  expanded  to  reveal  the  underlying  ratings  of  team 
performance.  The  scores  awarded  by  the  task  assessor  on  each  sub-goal  were  aggregated  to 
produce  the  average  for  each  team  goal  and  goal  category,  for  example,  the  average  of  all 
scores  underlying  Team  Goal  3.2:  M  anage Equipment  is  equal  to  3.4.  Unrated  sub-goals  were 
omi  tted  f  rom  th  i  s  cal  cu  I  ati  on .  P  ass/  f  ai  I  cri  ter  i  a  can  be  mod  i  f  i  ed  to  su  i  t  th  e  goal  s  of  an  exerci  se. 
In  EBS08,  sub-goals  on  which  performance  was  rated  as  lor  2  were  designated  fails,  as  were 
goalsand  goal  categorieswhereunderlying  scores  averaged  to  less  than  2.5.  In  addition,  any 
goal  or  goal  category  with  an  underlying  critical  fail  was  designated  a  fail  also.  Scores  for  sub¬ 
goals,  goals  and  goal  categories  were  colour  coded,  with  high  scores  appearing  in  green, 
moderate  scores  appeari  ng  i  n  yel  I  ow,  I  ow  scores  appeari  ng  i  n  orange,  and  scores  resulti  ng  i  n 
a  fail  appearing  in  red.  Sub-goals  that  were  not  rated  were  omitted  from  the  performance 
summary. 
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Figure9:  TheA  WAR  debrief  user  interface,  showing  the  A  BM  team's  performance  on  Vignettes  A 
and  B.  Goal  3.2  has  been  expanded  to  reveal  performance  ratings  on  ail  underlying  sub¬ 
goals. 


The  populated  A  WAR  tool  provides  access  to  the  electronic  scores  entered  by  an  assessor 
immediately  after  the  completion  of  a  mission.  This  data  can  be  used  to  guide  the  task 
performance  debrief,  with  colour  coding  drawing  the  attend  on  to  instances  of  exceptionally 
good  or  exceptionally  poor  performance  and  critical  fails.  Data  can  be  displayed  alongside 
data  from  previous  missions  for  comparison.  In  Figure  9,  team  performance  on  Vignette  A  is 
displayed  alongsideteam  performance  on  Vignette  B.  Figure9also  shows  that  comments  can 
be  viewed  by  floating  the  mouse  over  the  associated  score.  Graphs  can  be  produced  to 
compare  the  performance  of  a  single  team  between  missions,  or  of  two  or  more  teams 
completing  the  same  mission.  The  prompt  availability  and  electronic  format  of  scores 
contrasts  with  the  approach  currently  employed  by  the  RAAF,  where  scores  and  comments 
arehand  written  on  paper  forms.  Whi I ethese  notes  guidethe debrief,  no  scores  displayed  to 
the  team.  Scores  and  comments  are  typed  into  an  electronic  form  and  printed  for  team 
members  after  the  completion  of  the  mission.  Flowever,  this  provision  of  written  feedback 
may  takeweeks,  depending  on  theworkload  of  the  assessor,  by  which  time  team  member's 
memories  of  specific  aspects  of  the  mission  will  have  degraded  and  feedback  will  be  less 
effective.  The4  WAR  tool  allows  access  to  this  data  immediately  and  circumvents  thedouble 
handling  that  currently  occurs. 
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2.3.5  AAR  approach  and  tools 

At  the  cond  usi  on  of  each  E  BS08  mi  ssi  on,  the  A  BM  i  instructor  and  A  BD  debri  efed  parti  ci  pants 
on  their  task  work  and  teamwork  performance.  During  this  activity,  carried  out  as  an 
interactiveAAR  session  usingthetoolsinth  eADGESIM  suite,  assessors  highlighted  specific 
areas  for  improvement,  made  comparisons  between  the  mi  ssi  on  just  completed  and  previous 
missions,  and  outlined  specificinstanceswherebreakdownsin  doctrine,  communications,  and 
tactics  occurred. 

The  approach  taken  to  AAR  in  EBS08wastojuxtaposetwodifferentvisual  representations  of 
the  scenario  under  review.  The  first  representation,  known  as  'user  truth',  was  comprised  of 
elements  of  the  scenario  as  they  were  seen  by  individual  team  members,  and  the  resultant 
actions  taken  in  response  to  these  elements.  This  representation  is  distinctly  different  from  the 
second,  'ground  truth',  which  describes  what  actually  transpired,  insofar  as  capturing  the 
idealised  representation  of  entities  and  interactions  at  thefidelity  modelled  by  the  simulator 
itself.  This  juxtaposition  allowed  the  assessors  to  compare  what  was  happening  in  the 
simulation  with  what  the  users  were  seeing  and  their  consequent  reactions,  giving 
participants  the  opportunity  to  learn  lessons  about  their  actions  in  a  broader,  exercise-wide, 
context. 

I  n  ord  er  to  i  mp  I  ement  th  i  s  method ,  the  si  mu  I  ati  on  en  vi  ron  ment  w  as  conf  i  gu  red  w  i  th  a  su  bset 
of  tools  from  tbeADGESIM  suite,  designed  to  facilitate  si  mu  I  ati  on  recording  and  AAR.  This 
package  of  tools,  known  as  the TAA  RDIS  suite,  allows  an  AAR  activity  of  this  kind  to  take 
p  I  ace  i  n  real  -ti  me,  w  i  th  mi  n  i  mal  d  ow  nti  me  betw  een  th  e  en  d  of  si  mu  I  ati  on  an  d  the  beg  inning 
of  AAR.  The  sub-applications  of  which  TAARDIS  is  comprised  are:  Console,  Worldview, 
Pensieve  and  ReM  arks,  with  each  playing  a  specific  rolein  the  AAR  activity. 

The  main  component  of  TAARDIS,  the  Console,  primarily  allows  real-time  recording  and 
playback  of  scenario  network  data,  i  n  theform  of  Dl  S  Protocol  Data  U  nits  (PDU ).  These  PDU  s 
encapsulate  all  entity  activities,  weapon  interactions,  and  electromagnetic  transmissions 
(including  I FF,  weapons-lock,  and  radio  traffic),  as  well  as  additional  simulation  data  used  in 
modelling  and  scenario  representation.  This  stream  of  data  describes  exactly  what  data  was 
used  in  the  generation  of  the  scenario,  providing  a 'ground  truth'  for  the  exerci  se.  7  A  A  RDIS 
Console  al  lows  for  playback  of  this  data  in  real-ti  me,  or  at  speeds  slower  or  faster  than  real¬ 
time.  This  data  can  be  used  to  re-stimul  ate  simulation  components  in  order  to  regenerate  the 
scenario,  or  can  be  viewed  directly  using  tbeTAARDIS  Worldview  tool,  as  was  the  case  in 
EBS08.  Console  also  provides  for  remote  control  and  data  collection  for  other  TAARDIS 
components.  This  means  that  a  single  assessor  has  control  over  data  collection  and  replay  of 
an  entire  scenario. 

Replay  data  from  L4  A  RD/S  Console  is  interpreted  by  TAARDIS  Worldview,  and  displayed 
symbol  i  cal  I  y  on  a  three-d  i  mensi  onal  worl  d  map,  i  n  a  manner  consi  stent  with  other  A  D  G  ESI  M 
simulation  tools.  Additional  second  order  information,  such  as  targeting  and  weapons 
deployment  information,  is  also  displayed,  allowing  an  assessor  to  view  scenario  events  as 
they  actually  happened. 

The  corollary  to  this,  that  is,  what  the  user  actually  sees  (i.e.  'user  truth'),  is  captured  by 
TAARDIS  Pensieve.  Thistool  is  remotely  controlled  by  Console,  and  is  used  to  capture  the 
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screen  activities,  audio  input  and  audio  output  of  a  single  ABM  team  member's  console. 
When  deployed  across  all  work  stations,  it  allows  the  capture  of  'user  truth'  for  the  entire 
team.  For  AAR,  Pensieve  switches  from  recording  to  playback  mode,  and  shows  the  entire 
screen  recording  as  a  video  playback.  This  playback  is  controlled  remotely  by  Console, 
ensuring  that 'ground  truth'  and  'user  truth'  remain  in  synchrony,  allowing  viewers  to  make 
comparisons  between  the  two. 

Thefinal  el  ement  of  the  suite,  ReM  arks,  allows  an  assessor  to  insert  timestamped,  descriptive 
bookmarks  i  nto  the  Consol  e  record  i  ng  d  u  ri  ng  mi  ssi  on  executi  on.  These  bookmarks  hi  ghl  i  ght 
events  of  interest  for  later  review,  and  during  an  AAR  session  assessors  can  rapidly  skip  to 
these  bookmarks  to  emphasise  a  particular  lesson  or  learning  outcome. 

Given  thi  s  synchronous  recordi  ng,  bookmarki  ng,  and  replay  capabi  I  ity,  assessors  were  abl  eto 
accurately  represent  both  user  and  ground  truth  in  a  unified,  synchronous  manner.  During 
EBS08,  AAR  generally  involved  only  the  WD's  video,  displayed  on  Projection  2,  with 
complementary  ground  truth  being  shown  on  Projection  1  (see  Figure4).  This  side-by-side 
representati  on  of  data  was  val  uabl  e  duri  ng  the  acti  vity  as  it  al  I  owed  A  BM  team  members  to 
compare  their  perception  of  the  exercise  (as  shown  on  their  console),  with  the  events  that 
actual  I  y  occurred.  I  n  addition,  thetask  assessor  was  abl  eto  visual  ly  i  1 1  ustrateshortcomi  ngs  or 
events  of  interest  to  theteam  and,  by  using  bookmarks,  navi  gatefrom  one  event  of  interest  to 
the  next  easily  and  rapidly. 


3.  Participant  feedback  on  tools  and  techniques 

Overall,  the  feedback  from  EBS08  participants  was  very  positive,  with  all  participants 
bel  ieving  their  performance  at  BS08  would  benefit  from  the  additional  training  they  received 
at  EBS08  and  several  remarking  that  activities  of  this  kind  should  be  used  to  support 
preparation  for  all  I  arge  scale  live  exercises  involving  41WG  in  the  future.  Participants  also 
provided  feedback  on  each  ofthetoolsand  techniques  used  in  EBS08.  Overall,  feedback  was 
positive,  but  some  shortcomings  were  noted.  The  following  subsections  summarise  the 
information  provided  by  participants  and  the  observations  made  by  human  factors  scientists 
regard  i  ng  the  strengths  and  I  i  mi  tati  ons  of  thetool  s  and  techni  ques  eval  uated .  Each  subsecti  on 
concludes  with  a  list  of  recommendations  derived  from  this  information.  The  complete  list  of 
recommendations  is  included  in  Appendix  B. 

3.1  Th eADGESIM  simulation  system 

Thefol  I  owi  ng  secti  ons  outl  i  nethe  comments  made  by  the  parti  ci  pants  and  theobservati  ons  of 
human  factors  scientists  in  regard  toth  eADGESIM  simulation  system.  Thefeedback  of  the 
SIMOP  team  focussed  mainly  on  preparation,  capacity  and  coordination  issues,  and  the 
feedback  of  the  ABM  team  addressed  thefi  del  ity  of  thetraining  experience  they  received.  The 
feed  back  of  exerci  se  staff  general  I  y  rel  ated  to  the  performance  of  the  system  and  the  nu  mber 
of  operators  required  for  successful  implementation. 
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3.1.1  SI  MOP  feedback  on  theAD  GES/M  simulation  system 

Members  of  the  SI  MOP  team  reported  that  the  one-hour  familiarisation  session  on  the  new 
version  of  PSI  and  the  communications  system,  DISVOX ,  was  not  sufficient  given  the 
d  i  ff  erences  betw  een  these  tool  s  and  the  basel  in  eADGESIM  capabi  I  i  ty  w  i  th  w  hi  ch  they  w  ere 
more  familiar.  In  addition,  some  aspects  of  the  context  based  information  display  concept 
embod  i  ed  i  n  the  I  atest  versi  on  of  P  SI  were  not  compl  etel  y  val  i  dated  pri  or  to  EBS08.  D  ueto  the 
rel  ati  vel  y  br  i  ef  trai  n  i  n  g  p  rov  i  d  ed  on  D  ay  1  of  E  BS,  th  e  SI  M  0  P  s  w  ere  not  aw  are  of  al  I  avai  I  abl  e 
software  functions,  with  some  even  mistaking  unfamiliar  features  for  faults  or  vice  versa. 
Even  by  the  I  ast  day  of  the  exerci  se,  the  operators  were  not  usi  ng  these  systems  to  thei  r  ful  I 
potential  due  to  lack  of  familiarity  and  their  performance  was  affected  by  minor  software 
faults  or  defi  ci  enci  es.  The  SI  M  O  P  team  requested  that  a  peri  od  of  two  days  be  set  asi  de  pri  or 
to  theexerci  sefor  the  provi  si  on  of  trai  ni  ng  on  unfami  I  i  ar  systems  i  n  future.  They  bel  i  eved  that 
thiswould  improve  the  quality  of  training  provided  totheABM  team  through  a  reduction  in 
operator  error  and  inefficient  behaviour. 

The  workload  reported  by  the  SIMOPs  during  the  EBS08  scenario  vignettes  was  high,  as 
shown  in  Figure  10.  Figure  10  also  shows  a  considerable  difference  between  the  workload 
reported  by  SIMOPsand  that  reported  bytheABM  team,  suggesting  thattheSIMOPsfound  it 
difficult  to  create  a  sufficiently  challenging  environment  for  the  ABM  team.  Under  high 
workload  conditions  the  SIMOPs  were  observed  to  shed  a  number  of  tasks,  including 
maintaining  awareness  of  entity  altitude  and  speed,  announcing  the  future  intent  of  their 
entities  and  acknowledging  requests  from  team  members.  Specific  cognitive  aspects  of  these 
tasks,  and  the  relative  importance  of  the  tasks  themselves,  would  have  contributed  to  the 
likelihood  that  they  were  shed  ahead  of  other  tasks.  These  unfulfilled  tasks  reduced  the 
realism  of  the  training  scenario.  The  SI  MOP  team  attributed  a  large  proportion  of  their 
workload  to  the  number  of  entities  they  were  required  to  control.  In  a  SACTU  training 
context,  the  nu mber  of  enti  ti  es  assi  gned  to  an  i  nd i  vi  d ual  operator  i  s  not  I  i  kel  y  to  exceed  8,  yet 
at  EBS08this  number  was  at  times  greater  than  20.  The  SIMOPs  believed  that,  with  further 
improvement  and  automation,  PSI  could  supportthecoordinated  control  of  a  large  number  of 
entities.  Additional  workload  was  incurred  when  a  single  operator  was  assigned  multiple 
formations  with  varying  warfare  roles,  such  as  'strikers'  versus  'sweepers'.  These  roles  are 
distinguished  in  terms  of  aircraft  performance  constraints  as  well  as  timing  and  risk  related 
objectives;  and  the  Blue  SI  MOP  team  found  it  difficult  to  switch  between  the  mindsets 
required  to  perform  these  different  roles. 
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Figure  10:  The  self-reported  workload  of  the  SIM  OPs  and  ABM  team  during  the  vignettes  they 
experienced  over  thefivedays  ofEBS08 

The  Red  SI  MOP  team  reported  an  extremely  high  internal  communications  load.  They 
believed  that  this  was  due  to  a  lack  of  awareness  within  the  team  of  which  tasks  were 
performed  by  each  member.  Additional  communication  was  required  to  verify  entity 
ownership  and  the  radio  frequencies  monitored  by  each  SI  MOP.  The  RSC  was  sometimes 
unsu  re  of  w  hi  ch  SI  M  O  P  was  best  pi  aced  to  recei  ve  a  scrambl  ed  or  'regenerated '  formati  on .  A  s 
a  result,  workload  was  often  not  balanced  between  team  members.  Over  the  course  of  the 
exercise,  coordination  of  the  Red  SI  MOP  team  improved  via  discussion  of  a 
takeover/  handover  plan  after  each  vignette. 

3.1.2  ABM  feedback  on  training  fidelity 

M  embers  of  theA  BM  team  reported  thatthereweresomei  nstances  duri  ng  the  exercise  where 
they  were  requi  red  to  modify  thei  r  behavi  our  to  account  for  I  i  mitati  ons  i  n  the  fi  del  ity  of  the 
exercise.  These  limitations  fell  within  two  main  categories.  The  first  category  was  the 
perceived  fidelity  of  the  synthetic  C2  'picture'  provided  byth  eADGESIM  applications.  This 
representation  is  a  combination  of  the  qualities  of  sensor  outputs  and  the  behaviour  of 
detected  entities.  The  latter  is  confounded  by  both  the  performance  of  aircraft  model  sand  the 
application  of  tactics  by  SIM  OPs.  Second,  the  perceived  fidelity  of  communications  in  the 
simulation  system,  in  terms  of  the  intensity  and  complexity  of  information  exchange 
behaviours  and  the  constraints  of  the  medium. 

TheABM  team  reported  that  sensor  coverage  and  aircraft  mod  els  generally  produced  realistic 
detections.  However,  at  the  beginning  of  exercise,  a  SensorL/nk  fault  caused  the  IFF  code  of  a 
specifi  c  mode  that  was  assi  gned  to  a  gi  ven  ai  rcraft  to  be  reported  as  part  of  a  radar  contact  for 
another  ai  rcraft.  Under  exercise  conditions,  specific  IFF  modes  and  codes  are  used  to  identify 
aircraft  and  indicate  to  all  participants  when  aircraft  are  'out  of  exercise'.  The  fault  was 
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noti  ced  when  the  status  of  an  ai  rcraft  was  i  ncorrectly  decoded  as  'out  of  exerci  se',  confusi  ng 
the  ABM  team.  Fortunately,  thefault  occurred  only  atthe  beginning  and  was  rectified  for  the 
remainder  of  the  exercise  and  therefore  caused  little  disruption. 

While  the  EBS08  C2  picture  was  complex  relative  to  many  other  simulation  exercises,  the 
ABM  team  felt  it  was  somewhat  less  complex  than  live  exercises.  This  was  due  primarily  to 
constraints  imposed  by  the  size  of  the  White  Force.  Under  conditions  of  high  workload 
SIMOPs  attempted  to  reduce  the  complexity  of  their  control  task,  by  limiting  the  use  of 
altitude  changes  to  entire  formations.  This  resulted  in  a  scenario  presentation  that  was  more 
two-dimensional  than  usual.  In  live,  particularly  multinational,  exercises,  formations  may 
often  remain  within  a  single  safe  altitude  block  to  reduce  the  risk  of  collision.  Flowever,  in 
general  the  ABM  team  perceived  that  the  SIMOPs  did  not  have  the  same  instinctive 
appreciation  for  doctrine  and  hence  three-dimensional  geometry  as  pilots.  In  smaller  scale 
si  mul  ati  ons  SI  M  0  Ps  can  pi  ay  pi  I  ot  rol  es  convi  nci  ngl  y ,  yet  the  hi  gh  w  orkl  oad  experi  enced  i  n 
EBS08  reduced  theabilityofSIMOPsto'fly'  realistically.  The  ABM  team  did  not  believe  that  it 
would  be  feasible  to  train  SIMOPs  to  produce  the  desired  behaviours  under  such  high 
workload. 

The  ABM  team  reported  a  lack  of  realism  in  the  pattern  of  information  exchange  with  the 
simulated  pilots.  In  the  live  domain,  both  the  ABM  team  and  pilots  push  and  pull 
information.  During  EBS08theSIMOPsonly  pushed  a  subset  of  thetypical  information,  and 
occasionally  information  was  not  pulled  as  expected.  For  example,  SIMOPs  were unableto 
repl  i  cate  the  breadth  of  'fill  i  n'  communi  cati  ons,  such  as  those  from  pi  I  ots  shari  ng  i  nformati  on 
with  other  members  of  the  formation  and  theABM .  Similarly,  there  were  fewer  instances  of 
communications  being 'stepped  on',thatis,  multipleparti  es  attempting  to  transmiton  aradio 
channel  at  once.  SIMOPs  are  unableto  'step  on'  their  own  transmissions,  as  a  single  SI  MOP 
may  only  communi  cate  for  one  'pi  lot'  of  the  group  they  represent  at  a  ti  me.  The  reduction  of 
'fill  in'  communication  may  have  been  exacerbated  by  high  workload.  The  challenge  of 
maintaining  and  communicating  accurate  SA,  despite  the  constraints  of  a  real 
communications  system,  is  a  substantial  source  of  ABM  workload  and  hence  scenario 
i  ntensity  i  n  a  I  i  ve  envi  ronment.  The  A  BM  team  bel  i  eved  that  the  absence  of  some  of  these 
constraints  decreased  training  fidelity.  Specifically,  it  forced  theABM  team  to  pull  more 
information  from  the  SIMOPs  and  exercise  greater  control  over  their  actions  than  is  typical  in 
the  live  domain. 

TheABMscould  hear  communications  between  WhiteForce  members  and  often  waited  until 
theSIMOP  'flying'  the  ai  rcraft  they  needed  to  redirect  was  availableto  communicate.  This 
was  possible  because  theABM  team  was  physically  separated  from  theWhiteForce  only  by 
thin  partitions  that  did  not  reach  the  ceiling  and  will  be  resolved  in  future  exercises  if  the 
teams  are  I  ocated  i  n  separate  rooms.  I  n  a  I  i  veenvi  ronment,  A  BM  s  may  del  ay  communi  cati  ons 
to  a  pilot  in  their  zone  if  the  pilot  is  overheard  speaking  with  another  pilot  via  radio. 
Flowever,  the  additional  avenue  of  insight  into  pilot  communications  loads  that  arose  at 
EBS08  was  artificial  and  impacted  on  the  realism  of  the  simulation. 

The  ABM  team  also  stated  that  the  simulated  radio  coverage  provided  in  EBS08  was 
inconsistent  with  the  simulated  radar  coverage  and  did  not  degrade  in  any  realistic  way.  In 
the  live  domain,  and  if  ground  based  radio  and  radar  systems  are  co-located,  aircraft  that 
cannot  be  'seen'  are  not  generally  'heard'.  Furthermore,  the  loss  of  radio  coverage  may  be 
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gradual  such  that  either  the  ground  station  or  the  aircraft  may  still  receive  a  weakened  signal, 
even  i  f  thei  r  ow  n  transmi  ssi  ons  are  not  recei  ved .  A  I  oss  of  coverage  may  prompt  other  ai  rcraft 
to  relay  messages  between  ABM  s  and  pilots  who  do  not  have  direct  communication.  Such 
behaviour  did  notoccurin  EBS08,as  perfect  radio  communication  was  possi bleat  all  times. 

3.1.3  Exercisestaff  feedback  on  the  AD  GESIM  simulation  system 

PSI  and  its  related  server  components  suffered  several  major  and  minor  faults  through  the 
exerci  se.  These  w  ere  severe  on  thef  i  rst  day  of  the  exerci  se,  but  were  red  uced  over  the  cou  rse 
of  the  week.  Some  of  thefaults  were  rel  ated  to  new  fundi  onal  ity  added  just  pri  or  to  exerci  se 
commencement.  Future  exercises  should  utilise  mature  software  versions  and  prohibit  last 
minute  enhancements.  Adequate  time  must  beset  asideto  perform  scaled,  scenario-specific 
testing  to  ensure  that  all  technical  issues  are  resolved  before  the  exercise  begins.  Preparation 
adivities  should  also  include  the  development  of  a  human-machine  interface  and  team 
coordination  procedures  training  package  that  is  relevant  to  the  adivity.  This  should  be 
provided  to  the  SI  MO  Ps  in  clear  and  concise  manner  in  advance  of  the  exercise. 

Regardl  ess  of  enhancements  to  preparati  on  or  software  i  nterfaces,  the  avai  I  abl  e  White  Force 
personnel  were  insufficient  to  represent  the  number  and  nature  of  entities  required  by  the 
EBS08/  PB08  scenario  without  some  compromises  to  behavioural  fidelity.  Ideally  atleastfour 
Red  Force  and  th  ree  Bl  u  e  Force  SI  M  0  Ps,  as  w  el  I  as  one  experi  enced  mi  ssi  on  d  i  redor  assi  gned 
to  each  side  would  have  been  employed  as  the  White  Force.  Unfortunately,  all  of  theRAAF 
personnel  assigned  as  SI  MOPs  at  the  time  were  involved  in  the  exercise.  I  n  the  short  term, 
additional  SI  MOPs  are  required  to  meet  the  training  needs  of  similar  preparation  adivities. 
Thesecoul  d  be  either  recruited  by  the  RA  A  F,  or  contraded  through  ci  vi  I  i  an  servi  ce  provi  ders 
such  as  Milskil.  In  the  long  term,  additional  automation  is  required  within  computer 
generated  forces  packages  such  as  A  D  GESIM  to  reduce  SI  M  OP  workload.  Automation  should 
address  mechanisms  by  which  the  ratio  of  SI  M  OPs  to  exercise  participants  can  be  reduced, 
such  as  voice  recognition  and  synthetic  speech  technologies  to  allow  agents  to  control  entities. 

3.1.4  Recommendations: 

1.  All  simulator-related  technical  briefsshould  bedearand  contain  theappropriatelevel  of 
detail. 

2.  SIMOPs  should  receive  adequate  training  in  the  use  of  all  software  and  simulation 
systems  before  the  commencement  of  an  exercise.  The  SI  M  OP  team  suggested  that  two 
days  of  focused  training  in  the  week  prior  to  EBS08  would  have  permitted  them  to 
become  sufficiently  familiar  with  the  new  versions  of  PSI  and  DISVOX . 

3.  Adequatetimeshould  bereserved  intheweeksprecedinganexerciseforathoroughand 
systematic  testing  of  ADGESIM  systems.  All  technical  issues  must  beresolved  priortothe 
exercise  with  no  additions  or  modifications  made  after  testing. 

4.  SI  M  0  P  w  orkl  oad  w  i  1 1  be  consi  d  erabl  y  red  u  ced  i  f  enti  ti  es  are  assi  gned  by  rol  e  rather  than 
location.  White  Force  scenario  managers  should  endeavour  to  achieve  this  wherever 
possi  ble. 
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5.  Further  workload  reduction  can  be  achieved  by  increasing  the  number  of  SIMOPs 
assigned  to  the  White  Force.  Flowever,  all  qualified  and  partially  qualified  RAAF  SIMOPs 
were  present  at  EBS08  and  the  pool  of  avai  I  able  SI  M  OPs  must  therefore  be  increased. 

6.  Th  eADGESIM  simulation  suite  should  be  further  automated  with  the  goal  of  further 
reducing  the  number  of  SIMOPs  required  to  produce  a  complex  scenario.  Flowever, 
modifications  must  be  sufficiently  tested  to  ensure  that  they  are  not  made  at  the  expense 
of  realism. 

7.  The  ABM  team  was  able  to  hear  the  activity  of  the  SIMOPs  and  adjusted  their 
communications  accordingly.  Ideally,  SIMOPs  and  ABM  teams  should  operate  in 
completely  separate  rooms  rather  than  partitioned  sections  of  the  same  room. 

3.2  Toteboard 

3.2.1  General  comments 

Th  eT  oteboard  was  ori  gi  nal  ly  dev  el  oped  at  SA  CTU  for  use  i  n  trai  ni  ng  events  conducted  i  n  the 
SACTU  simulator,  but  has  since  been  more  broadly  utilised.  41WG  ABMs  have  used  the 
T oteboard  i  n  al  I  recent  major  exerci ses  i  nvol  vi  ng  thei  r  parti ci pation,  i  nd  udi  ng  Pitch  Black,  A ces 
North,  Aces  South  and  the  East  Coast  Air  Defence  Exercise  (ECADEX).  All  ABM  team 
members  had  therefore  been  exposed  to  the  Toteboard,  either  through  a  SACTU  simulation 
exercise  or  a  live  exercise  at  3CRU,  prior  to  participating  in  EBS08. 

TheABM  team  f  el  t  th  at  th  eT  oteboard  provided  all  of  the  information  they  required  during 
mission  execution.  An  additional  benefitwasthattheWFM  D  was  ableto  view  the  Toteboard 
during  missions.  This  provided  an  effective  means  for  coordination  between  the  tactical 
control  centreand  the  White  Forcecontrol  centre,  for  example  when  launching  formations.  In 
these  instances,  an  ABM  team  member  would  launch  aformation  and  annotateit  accordingly 
on  the  Toteboard.  TheWFMD  would  receive  the  request  and  action  it  through  coordination 
with  the  SIM  OPs. 

The  RAAF  participants  did,  however,  offer  two  suggestions  for  improvement.  Firstly,  they 
suggested  that  the  accuracy  of  enduranceesti  mates  be  i  mproved.  Theversi  on  of  the!  oteboard 
used  in  EBS08  calculates  endurance,  that  is,  airborne  time  remaining  before  the  aircraft  runs 
out  of  fuel,  by  subtracting  the  number  of  minutes  an  aircraft  has  been  airborne  from  the 
number  of  minutes  that  type  of  aircraft  could  be  expected  to  operate  on  a  full  fuel  load. The 
ABM  team  suggested  that  the  calculation  would  be  much  mo  re  accurate  if  it  also  considered 
weather,  airfield  conditions  and  fuel  burn  rate.  They  also  requested  an  esti  mate  of  bin  go  fuel, 
that  i  s,  the  ti  me  after  w  hi  ch  the  ai  rcraft  w  i  1 1  not  have  suffi  ci  ent  fuel  to  retu  rn  from  i  ts  cu  rrent 
location  to  home  base.  Secondly,  the  Toteboard  should  be  protected  from  corruption  due  to 
competing  user  inputs.  The  Toteboard  was  a  shared  spreadsheet  and  information  could 
therefore  be  updated  by  multi  pie  users.  Flowever,  iftwo  users  concurrently  updated  thesame 
piece  of  information,  the  T oteboard  malfunctioned,  causing  the  information  to  be  lost,  the 
spreadsheet  to  be  locked  prohibiting  the  entry  of  further  information  or  the  system  to  crash. 
The  Toteboard  has  since  been  upgraded  in  linewith  these  recommendations. 
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3.2.2  Recommendations 

1.  The  most  recent  version  of  the  Toteboard  should  beusedforall  futureABM  simulation 
exercises. 

3.3  T he TDT framework 

3.3.1  General  comments 

Theinitial  reaction  of  the  ABM  teamtothe7D7  concept  was  positive.  However,  they  did  not 
find  the7D  7  i  ntroductory  exercise  (refer  A  ppendixC)  valuable.  They  feltthatthecontext  and 
behaviours  displayed  by  personnel  described  in  the  written  summary  of  the  Blackhawk 
accident  were  extreme  and  were  not  sufficiently  aligned  with  the  ABM  team's  own  work 
environment.  As  a  result,  they  reported  that  they  did  not  believe  the  introductory  TDT 
exercise  improved  their  ability  to  recognise  subtle  examples  of  good  and  bad  teamwork 
behaviour.  This  negative  appraisal  may  have  had  an  impact  on  the  team's  participation  in 
TDT  discussions  and  on  their  motivation  to  attend  to  teamwork  factors  during  the  exercise. 

A  requirement  of  the  TDT  approach  is  that  the  workload  of  the  participating  team  be 
sufficiently  high,  with  sufficient  strain  placed  on  communication  and  coordination  between 
team  members.  U  nfortunately,  A  BM  team  workload  was  lower  than  i  ntended  dueto  system 
instability  and  theSIMOPs  lack  of  familiarity  with-4  DGESIM  .  Asaresult,theEBS08vignettes 
d  i  d  not  pi  ace  a  great  amount  of  strai  n  on  teamwork  behavi  ours  and  the  A  BD  struggl  ed  to  fi  nd 
examples  of  behaviour  that  could  beimproved.Thisfurtherimpactedonthemotivationofthe 
ABM  team  to  engage  in  the  TDT  portion  of  the  debriefs. 

Smith-Jentsch,  et  al.  (1998)  suggest  that  teamwork  performance  be  evaluated  by  four  TDT 
assessors,  with  one  allocated  to  each  TDT  category.  This  arrangement  was  not  possibledueto 
personnel  constraints  and  theABD  was  therefore  tasked  with  assessing  performance  within 
all  four  categories.  According  to  Smith-Jentsch,  et  al.,  this  is  an  acceptable  situation  if  the 
assessor  sel  ected  has  extensi  ve  experi  ence  in  TDT  assessment.  Therefore,  the  TDT  aspects  of 
the  EBS08  assessment  and  feedback  may  havebeen  improved  had  theABD  been  given  greater 
opportunity  to  becomefami  I  i  ar  w  ith  theapproach  prior  to  the  commencement  of  the  exercise. 

The  rol  e  of  thefaci  I  i  tator  duri  ng  a  TD  7  debri  ef  i  s  to  gui  de  the  team  through  a  process  of  self- 
evaluation  and  correction.  At  EBS08,  this  rolewas  not  fully  implemented,  with  theABD  often 
providing  direct  feed  back,  rather  than  facilitation.  The  ABM  teamdid  report  that  there  were 
instances  where  the  7D  7  debrief  stimulated  discussions  not  raised  in  the  task  performance 
debrief.  However,  the  largely  uni-directional  nature  of  feedback  may  have  reduced  the 
motivation  of  theteam  members  and  is  likely  to  have  reduced  benefits  to  teamwork.  If  7D  7  is 
to  be  used  to  structure  assessment  and  feedback  in  future  exercises,  assessors  should  be 
provided  with  training  that  improves  their  understanding  of  the  framework  itself  and  of  the 
i  importance  of  thefaci  I  itatory  styleon  which  it  is  based. 

The  A  BM  team  reported  that  al  I  four  of  the7  D  7  di  mensi  ons  werefami  I  i  ar  constructs  and  that 
they  were  reflected  in  the  performance  assessment  approach  currently  employed  withinthe 
RAAF  but  were  more  implicitly  than  explicitly  trained.  TheABD  felt  that  the  issues  raised  in 
his  7D7  debriefs  would  have  been  implicitly  covered  in  a  standard  debrief,  regardless  of 


27 


DSTO-TR-2305 


whether  he  had  been  introduced  to  the  TDT  procedure,  and  the  ABM  team  felt  that  an 
integrated  debrief  would  be  less  disjointed  and  more  effective.  The  ABM  team  did  notfeel 
that  theTD  T  framework  compl  emented  theculture  withi  n  the  A  ustral  i  an  mi  I  itary,  whi  ch  was 
described  as  having  less  emphasis  on  rigid  doctrine  and  structured  training  than  the  US 
military.  This  reflected  the  overall  sentiment  of  participants  that  theTD  7  process,  as  it  was 
implemented  at  EBS08,  offered  I  ittlemeansfor  the  improvement  of  teamwork  processes.  The 
ABM  team  felt  that  the  TDT  process  would  be  more  appropriate  for  less  experienced 
personnel  w  ho  had  not  yet  grasped  theteamwork  concepts  i  mpl  i  cit  i  n  trai  ni  ng.  They  bel  i  eved 
itwould  be  of  particular  use  if  its  implementation  resulted  in  the  introduction  of  acommon 
parlance  with  which  to  discuss  issues  of  team  behaviour. 

Thereisconsiderableevidenceto  suggest  that  training  focusing  on  team  processes  can  lead  to 
improved  team  performance (Kozlowski  &  llgen,  2006).Thereisalsoconsiderablesupportfor 
the  TDT  framework  as  an  effective  means  by  w  hi  ch  to  trai  n  teams  to  work  together  better,  at 
least  within  the  context  of  the  US  Navy  (Smith-Jentsch,  et  al.,  1998).  At  EBS08,  the 
i  mpl  ementati  on  of  TD  T  was  only  parti  al  and  thereforecond  usi  ons  regardi  ng  the  utility  of  the 
approach  cannot  be  drawn.  Whatwas  clear,  however,  was  the  resistance  of  the  RAAF  ABM 
team  to  theapproach.  For  approaches  ai  med  specifically  at  i  mprovi  ng  teamwork  processes  to 
be  effecti  ve  i  n  f  utu  re  exerci  ses,  they  must  be  accepted  by  parti  ci  pati  ng  teams.  Thi  s  w  i  1 1  requ  i  re 
an  investigation  of  potential  modifications  to  bring  these  approaches  into  line  with  RAAF 
culture. 

3.3.2  Recommendations 

1.  If  a  TDT  introductory  exercise  is  used  to  train  a  team  to  recognise  instances  of  the  four 
TD  T  el  ements,  care  shou  I  d  betaken  w  hen  sel  ecti  ng  the  seen  ari  o  descri  pti  on  to  ensure  that 
it  bears  relevance  to  the  context  within  which  the  team  usually  works.  The  descri  pti  on 
should  include  realistic  examples  of  good  and  bad  behaviour  as  well  as  extreme  examples 
included  to  maintain  interest.  Learning  to  recognise  these  subtle  examples  will  better 
equi  p  the  team  to  recogni  se  such  i  nstances  i  n  thei  r  own  performance. 

2.  The  scenario  workload  must  be  high  enough  to  genuinely  test  teamwork  and 
communication  skills. 

3.  I  fTD  T  isto  be  included  in  futu  reexercises,  assessment  should  becompleted  byfourTDT 
assessors,  with  oneassessing  performanceunder  each  TDT  category.  If  this  is  not  possible 
dueto  personnel  constraints,  the  assessors  that  are  involved  should  undergo  extensive 
trai  ni  ng  i  n  TD  T  assessment  pri  or  to  the  commencement  of  the  exerci  se. 

4.  FutureTDT  assessors  shou  Id  undergotrainingthatimprovestheir  understanding  of  the 
importance  of  guiding  the  team  through  a  process  of  self-evaluation  and  correction, 
rather  than  providing  direct  and  unidirectional  feedback,  and  their  ski  Ms  in  encouraging 
team  member  participation. 

5.  The  ABM  team  reported  that  the  structure  inherent  to  TDT  did  not  compliment 
Australian  military  culture.  Modifications  aimed  at  improving  the  suitability  of  TDT 
wi  thi  n  the  context  of  the  A  ustral  i  an  mi  I  itary  shoul  d  therefore  be  consi  dered . 
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3.4  The  AW  AR  tool 

3.4.1  Comparison  with  current  RAAF  assessment  approach 

Use  of  the  team  goal  hierarchy  forced  the  RAAF  instructor  assessing  task  performance  to 
consider  a  wider  range  of  performance  aspects  in  greater  detail  than  would  be  the  case  using 
the  assessment  system  currently  employed  by  the  RAAF.  The  ABM  team  felt  that  the  team 
goal  hierarchy  permitted  performance  to  be  assessed  more  objectively,  as  the  sub- goals  were 
more  specific  than  the  measures  currently  used.  H  owever,  they  felt  that  the  sub-goals  were 
still  too  broad,  leaving  them  open  to  interpretation.  It  was  suggested  that  performance  could 
be  more  objectively  assessed  if  the4  WAR  tool  was  populated  with  a  longer  list  of  finer- 
grained  measures,  and  an  assessor  was  asked  to  rate  performance  as  C  or  N  YC,  rather  than  on 
a  five-point  scale.  Due  to  the  complexity  of  the  environment  within  which  an  ABM  team 
operates,  such  a  list  would  need  to  be  extremely  extensiveand  detai  led.  Assessors  areal  ready 
under  considerable  time  pressure  and  could  not  commit  the  additional  time  required  to 
evaluate  performance  on  a  still  greater  number  of  measures.  The  ABM  team  and  the  task 
assessorfeltthattheratingsof  'terrible', ' poor ',  'adequate',  'good'  and  'excellent'  were  subjective. 
H  owever,  these  rati  ng  I  abel  s  were  chosen  because  research  suggests  that  they  attract  greater 
agreement  between  assessors  than  a  purely  numerical  scale  (Whitley,  1996).  In  addition,  a 
f  i  ve-poi  nt  rati  ng  seal  e  w  i  th  meani  ngful  and  quantifi  abl  e  rati  ng  I  abel  s  provi  des  greater  i  nsi  ght 
into  how  much  improvement  is  required.  If  a  five-point  rating  scale  is  to  be  used,  the  ABM 
team  suggested  that  a  detailed  description  of  each  of  the  five  performance  ratings  on  each 
measure  be  included.  While  this  may  promote  the  reliability  and  validity  of  performance 
measu  rement,  th  e  amou  nt  of  SM  E  i  n  p  u  t  th  at  w  ou  I  d  be  req  u  i  red  to  cl  ear  I  y  d  escr  i  be  beh  avi  ou  rs 
characteristic  of  performance  for  each  rating  of  each  measure  would  be  prohibitive.  It  is  not 
clear  whether  the  best  approach  to  assessment  i  s  for  assessors  to  rate  team  performance  as  C 
or  NYC  on  a  long,  fine-grained  list  of  measures  or  to  rate  performance  on  a  shorter  list  of 
broader  measures  usi ng  a  fi  ve-poi  nt  rati  ng  seal e.  It  does  seem,  however,  that  a  shorter  list  of 
broader  measures  would  be  more  resi  I  i  ent  if  the  two  approaches  imperfectly  applied  dueto 
ti  me  con strai  nts.  F  u  rth er  research  i  s  n eed  ed  to  d  eter mi  n e  th e  ap p roach  most  I  i  kel  y  to  p rod  u ce 
consistent  ratings  between  assessors  and  over  ti  me  given  the  constrai  nts  within  which  RAAF 
training  takes  place. 

The  RAAF  participants  felt  that  the  team  goal  hierarchy  would  be  extremely  useful  for 
designing  basiccoursescenariosatSACTU  andtheSchool  of  Air  Warfare(SAW).Thecurrent 
process  i  nvol  ves  the  assessors  i  nserti  ng  events  i  nto  a  scenari  o  that  they  bel  i  eve  wi  1 1  provi  de 
students  with  a  valuable  training  experience  and  theopportunitytoshowcasetheirskills.  The 
method  i  s  not  structu  red ,  rel  y  i  ng  enti  rel  y  on  the  experti  se  of  assessors,  and  the  events  i  nserted 
and  trai  ni  ng  received  i  s  therefore  I  i  kel  y  to  vary  between  assessors  and  over  ti  me.  I  n  additi  on, 
scenarios  are  usually  created  under  time  pressure  just  prior  to  their  use.  Using  a  structured 
tool  like  the  team  goal  hierarchy  to  design  scenarios  would  promote  consistency  between 
students  in  terms  of  their  exposure  to  events.  Using  the  same  scenario  for  all  students  at  a 
courselevel  would  permit  valid  performance  comparisons  between  students. 
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3.4.2  Performance  assessment 

The  RA  A  F  i  nstructor  pi  ay  i  ng  the  rol  e  of  task  assessor  w  as  not  abl  e  to  rate  performance  on  al  I 
measureswhilemissionswerein  progress  and  therefore  completed  performance  assessments 
after  thei  r  concl  usi  on.  The  reasons  given  for  this  were  two-fol  d .  Fi  rstly,  some  measures  were 
considered  to  relate  to  holistic,  rather  than  specific,  aspects  of  performance  and  therefore 
could  not  be  rated  until  after  the  conclusion  of  the  mission.  The  utility  of  the  team  goal 
hi  erarchy  rel  i  es  on  the  specif  i  ci  ty  of  measu  res,  and  thi  s  comment  therefore  suggests  the  need 
to  revise  some  measures.  Secondly,  although  thetask  assessor  found  the4  WAR  tool  simple 
and  i  ntui  ti  ve,  he  was  not  suffi  ci  entl  y  fami  I  i  ar  w  i  th  the  arrangement  of  goal  s  and  su  b-goal  s  i  n 
theteam  goal  hierarchy  and  therefore  spent  a  considerable  amount  of  ti  mesearchi  ng  for  sub¬ 
goal  s.  A  ssessors  at  SA  CTU  and  the  SA  W )  w  ou  I  d  be  si  mi  I  ar  I  y  u  nfami  I  i  ar  w  i  th  the  team  goal 
hierarchy,  and  would  require  suffi  dent  training  if  it  was  to  be  adopted  by  these  groups.  In 
addition  to  promoting  familiarity  with  theteam  goal  hierarchy,  such  training  should  aim  to 
develop  skills  in  assessing  entire  teams  rather  than  individual  team  members. 

3.4.3  Performance  feed  back 

The4  WAR  tool  debrief  user  interface  was  designed  so  that  only  one  goal  category  at  a  ti  me 
could  be  expanded  to  reveal  its  constituent  goals  and  sub-goals.  This  was  done  in  order  to 
minimise  distraction  from  scores  that  were  not  related  to  the  current  discussion.  The  ABM 
team  found  this  arrangement  effective  and  intuitive,  but  felt  that  it  drew  unwarranted 
attention  to  the  six  goal  category  averages.  These  averages  did  not  provide  information  or 
direction  on  how  to  improveand  could  therefore  be  seen,  themselves,  as  a  distraction.  It  was 
suggested  thattheformat  be  modified  to  display  all  sub-goals  and  accompanying  scores,  with 
the  assessor  scrolling  to  those  of  relevance  as  they  are  mentioned  in  thetask  performance 
debrief.  Perhapstheelementswithinthegoal  category  or  goal  of  interest  could  behighlighted 
with  all  others  visually  faded.  As  opinions  were  mixed  regarding  the  comparative  utility  of 
these  options,  the  solution  may  be  to  display  collapsed  goal  categories  on  the  debrief  user 
interface,  but  to  provide  students  with  a  printout  of  all  scores  and  comments  for  reference 
during  thetask  performance  debrief.  The  colour  coding  of  goal  category  averages  in  the 
debrief  user  interface  was  highly  valued  as  it  drew  attention  to  underlying  low  scores  and 
critical  fails.  The  scores  and  comments  relating  to  each  sub-goal  were  extremely  found  to  be 
useful,  as  they  provided  the  team  with  an  understanding  of  their  current  performance, 
showed  how  much  room  existed  for  improvement,  and  provided  instructionsfor  improving. 

There  are  a  number  of  advantages  to  stori  ng  scores  and  comments  i  n  an  electronic  format.  The 
el  ectronic  format  permits  thegraphical  or  tabular  comparison  of  scores  achieved  by  different 
teams  or  by  thesameteam  on  different  missions.  This  permits  theeval  uation  of  performance 
differences  between  teams  that  have  experienced  different  training  conditions  or  the 
eval  uati  on  of  performance  i  improvement  over  ti  mefor  a  si  ngl  e  grou  p.  The  A  BM  team  val  ued 
theabi  I  ity  to  display  their  scores  at  the  beginning  of  theweek  alongsidetheir  scores  at  theend 
of  the  week  to  determinethe  sub-goal  son  which  they  improved  over  the  course  of  theweek. 
As  stated,  the  ABM  teamdid  not  find  theaveragescoreachieved  on  each  goal  category  useful, 
and  thereforedid  not  find  ituseful  to  graph  these  averages.  Itmay  be,  however,  that  graphs 
that  permit  more  meaningful  comparisons,  for  example,  performance  differences  on  specific 
measures  between  teams  or  between  instances  for  the  same  team,  may  be  viewed  more 
positively.  In  addition,  information  from  the  electronic  database  can  be  used  to  improve 
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RAAF  training  courses.  Anonymous  data  could  be  compiled  and  analysed  to  indicate  the 
specific  tasks  that  RAAF  students  at  certain  qualification  levels  find  most  difficult,  and 
training  could  be  modified  to  improve  performance  on  these  tasks.  Note  that  this  possibility 
exists  only  if  performance  is  always  assessed  using  the  same  performance  measures. 

Enteri  ng  performance  assessment  scores  el  ectron  i  cal  I  y  from  the  outset  ci  rcu  mvents  the  dou  bl  e 
handling  that  occurs  when  paper  and  pencil  performance  ratings  are  transferred  into  a 
database.  The  current  RAAF  practice  is  for  paper  and  pencil  assessments  to  be  entered  into  a 
database,  printed  and  provided  to  students  as  soon  as  possible  after  the  completion  of  a 
mission.  Students  then  refer  to  this  report  in  their  own  time.  Flowever,  during  busy  periods, 
providing  written  feedback  can  take  up  to  a  week,  after  which  time  comments  and 
suggestions  are  far  less  effective  as  students  may  have  trouble  remembering  the  specific 
details  of  the  relevant  mission.  If  reports  could  be  provided  to  students  within  minutes  of 
completing  a  mission,  and  referred  to  during  debrief,  student  learning  is  likely  to  improve 
considerably.  The  current  version  of  >A  WAR  doesnotindudeafunctionthatallowsthelistof 
sub-goalsand  associated  performance  ratings  to  be  printed.  Flowever,  theimportanceofthis 
feature  has  been  recognised  and  itwill  therefore  beafeaturethatissoughtintoolsofthiskind 
for  future  exercises.  Itwas  suggested  that  assessors  should  have  the  ability  to  export  scores 
and  comments  to  a  spreadsheet  either  under  the  goal  categories  listed  in  the  team  goal 
hierarchy  or  in  the  order  of  timestamps  attached  to  assessor  comments.  The  RAAF 
parti  ci  pants  suggested  that  the  performance  report  i  ncl  udethreeto  five  good  aspects  of  thei  r 
performance  and  three  to  five  areas  for  improvement.  They  also  suggested  that  it  may  be 
possible  for  basic  course  students  to  debrief  themselves  after  the  completion  of  a  mission 
using  the  performance  report  and  video  and  audio  recordings.  This  would  be  particularly 
benef  i  ci  al  w  h en  th ere  are  very  few  i  n str u  ctors  w  i  th  great  d eman d s  on  th ei  r  ti  me,  as  i  s  th e  case 
at  SAW.  The  benefits  and  drawbacks  of  this  approach,  in  terms  of  student  learning,  would 
need  to  be  thoroughly  evaluated  before  implementing  this  approach. 

The  ABM  team  suggested  that  if  a  tool  I  i  ke/\  WAR  is  to  be  used  by  an  ABD,  efforts  should  be 
madeto  i  ncrease  its  portabi  I  ity,  as  A  BDs  often  rate  performance  whi  lestandi  ng  behi  nd  A  BM  s. 
Consideration  of  very  lightweight  tablet  PC  with  handwriting  recognition  software  is 
therefore  required.  If  handwriting  recognition  software  is  employed,  it  should  permit 
assessors  to  store  commonly  used  symbols,  shorthand  and  abbreviations.  Participants  also 
suggested  that  more  space  be  provided  for  assessors  to  comment  on  performance  and  be  able 
to  draw  freehand  diagrams  to  support  thecomments  made.  Thetask  assessor  suggested  that 
the  utility  of  the  tool  could  be  improved  through  greater  automation.  Fie  suggested  the 
i  ncl  usi  on  of  a  feature  permitti  ng  an  assessor  conducti  ng  a  task  performance  d  ebri  ef  to  cl  i  ck  on 
a  timestamp  next  to  a  comment  recorded  in  4  WA  R  to  cue  all  audio  and  video  recordings  to 
that  pointinthemissi  on.  This  would  eliminate  the  need  for  separate  bookmarking  and  permit 
the  assessor  to  focus  on  i  mprovi  ng  the  qual  ity  of  comments. 
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3.4.4  Recommendations 

1.  Assessment  should  involve  the  evaluation  of  performance  on  a  pre-determined  list  of 
measures  that  covers  al  I  relevant  aspects  of  performance,  such  at  the  A  BM  DC  A  team 
goal  hierarchy. 

2.  Performance  ratings  will  be  more  objective  and  consistent  if  measures  refer  to  specific 
aspects  of  performance,  are  clearly  worded  and  are  unambiguous.  Although  the  ABM 
DCA  team  goal  hierarchy  offers  an  improvement  on  the  current  RAAF  assessment 
approach,  further  refi nement  is  requi red.  Further  research  will  berequi red  to  i nform  such 
improvements. 

3.  Structuring  training  scenarios  around  the  elements  of  the  team  goal  hierarchy  would 
promote  consistency  between  teams  in  terms  of  their  exposure  to  events  and  resultant 
learning. 

4.  Using  the  same  scenarios  for  all  students  at  a  particular  course  level  would  permit 
reliable  performance  comparisons  to  be  made  between  students. 

5.  Assessors  must  be  sufficiently  trained  in  the  use  of  tools  such  as  AW  A  R  and  the 
arrangement  of  elements  in  the  incorporated  hierarchy  before  using  them  to  assess 
performance. 

6.  Electronic  databases  of  student  performance  data  may  draw  attention  to  tasks  that 
stu  d  ents  f  i  n  d  parti  cu  I  ar  I  y  d  i  ff  i  cu  1 1  an  d  trai  n  i  n  g  may  be  mod  i  f  i  ed  to  ad  d  ress  this.Thisand 
other  potential  uses  of  electronic  performance  data  should  be  considered. 

7.  Tools  like  A 1/1/4  R  allow  performance  assessment  data  to  be  entered  straight  into  an 
electronic  database,  circumventing  the  double  handling  that  currently  occurs  in  the 
RAAF  when  paper  and  pencil  performance  ratings  are  transferred  into  a  database. 

8.  A  function  should  be  incorporated  into  performanceassessmenttoolsthatpermitsscores 
to  be  exported  to  a  spreadsheet  and  printed  in  a  user-specified  order.  This  performance 
report  should  include  three  to  five  good  aspects  of  thei  r  performance  and  three  to  five 
suggestions  for  improvement  and  should  be  given  to  students  prior  to  the  debrief. 

9.  The  benefits  and  drawbacks  of  creati  ng  a  process  for  students  to  usea  performance  report 
and  audioand  video  rep  I  ay  sofa  mission  to  debrief  themselves  should  be  considered. 

10.  Lightweight  tablet  PCs  with  handwriting  recognition  should  be  considered  to  improve 
the  portability  of  software  tools  such  as  A 1/1/4  R .  Flandwriting  recognition  software  should 
permit  assessors  to  store  commonly  used  symbols,  shorthand  and  abbreviations. 

11.  The4l/1/4R  spreadsheet  should  provide  more  space  for  assessor  comments  and  should 
permit  freehand  diagrams  to  support  assessor  comments. 
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3.5  AAR  approach  and  tools 

3.5.1  Collecting  data  for  an  AAR 

TheABM  task  assessor  was  tasked  with  provi  ding  team  task  assessments  vi  a  the/\  WAR  tool 
and  with  bookmarking  key  scenario  events  via  theTAARDIS  Consoleapplication.  Hefound  it 
extremely  difficult  to  conduct  both  of  these  tasks  simultaneously  due  to  workload,  and 
suggested  that  the  tasks  be  allocated  to  two  separate  task  assessors.  H  owever,  as  the  key 
scenario  events  may  also  represent  evidence  for  assessments,  these  tasks  may  be  more 
effectively  and  efficiently  completed  by  one  person.  An  alternative  is  for  assessors  to  create 
timestamped  bookmarks  in  A  WAR  and  to  export  these  to  other  applications  (such  as 
TAARDIS).  Import  and  export  functionality  must  beset  up  to  support  this  capability. 

The  task  assessor  felt  that  monitoring  the  communication  of  specific  team  members  would 
improve  thequality  of  his  team  performance  ratings.  U  nfortunately,  dueto  a  limitation  of  the 
task  assessor's  headset,  the  relative  volume  of  the  communications  system  was  less  than  the 
general  noise  level  of  the  room.  On  occasion  he  needed  to  leave  his  workstation  and  stand 
behind  the  team  member  he  wished  to  listen  to.  This  meant  that  the  task  assessor  was  not 
alw  ays  able  to  enter  comments  into  either WAR  orTAARDIS  at  this  time,  and  had  to  update 
these  upon  his  return.  This  was  likely  to  have  reduced  the  clarity  with  which  the  assessor 
remembered  i  mportant  aspects  of  the  mission  and  increased  the  chance  that  some  would  be 
forgotten  entirely.  In  order  to  improve  flexibility,  future  exercises  should  investigate  the 
utility  of  completing/!  1/1/ AR  assessments  and  creating  bookmarks  using  a  tablet  PC. 

3.5.2  Prepari  ng  for  an  A  A  R 

AAR  sessi  ons  shoul  d  occu r  as  soon  as  possi  bl  e  after  the  compl  eti  on  of  each  vi  gnette  i  n  order 
to  promote  thetrai  ni  ng  benefit  of  feedback.  I  n  EBS08,  some  measures  requi  red  theeval  uation 
of  performance  across  the  entire  vignette  and  ratings  were  therefore  finalised  after  the 
compl  eti  on  of  vi  gnettes.  Thi  s  general  ly  del  ayed  the  A  A  R  sessi  on  by  a  few  mi  nutes.  The  A  A  R 
session  was  also  delayed  by  the  need  for  technical  staff  to  synchronise  thePens/eve  videos  and 
Consol  eground  truth  recording,  and  to  test  the  levels  of  the  rep  I  ay  audio.  Although  thedel  ay 
was  not  considered  critically  long,  it  was  unnecessary  and  the  tools  should  be  further 
modified  to  automate  this  synchronisation.  The  task  assessor  felt  that  the  AAR  tools, 
particularly  the  bookmarking  feature,  significantly  reduced  the  preparation  time  required 
prior  to  the  provision  of  feedback. 

3.5.3  Conducting  an  AAR 

The  task  assessor  felt  that  the  AAR  tools  facilitated  the  identification  of  key  points  and 
permitted  the  displaying  of  concise  examples  of  specific  behaviours.  The  task  assessor 
typically  selected  the  ground  truth  and  WD  views  for  projected  display  during  theAAR. The 
WD  view  was  chosen  because  the  view  selected  by  the  WD  was  typically  wider  than  that 
selected  by  the  other  team  members.  ABM  team  members  stated  a  preference  for  short 
mission  reviews  that  id  entity  and  emphasisekey  learning  points.  They  felt  that  the  rep  I  ays  of 
the  scenario  ground  truth,  the  individual  console  video  and  associated  communication 
recordings  were  an  effective  and  efficient  means  of  illustrating  learning  points.  The 
synchronisation  between  recordings  and  ability  to  cueto  relevant  bookmarks  increased  the 
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efficiency  of  AAR  sessions.  However,  therewas  a  synchronisation  problem  associated  with 
seeking  to  a  bookmark.  The  Pens/eve  video  would  seek  and  begin  playback  immediately, 
whi  1st  the  1/1/ orldV  iew  map  di  spl  ay  would  show  the  history  trai  I  of  entities  i  n  thei  r  'pre-seek' 
position  and  gradually  update  the  position  of  entities  over  thefollowing  five  seconds.  This 
distracted  thetraining  audience  and  interrupted  theflow  of  theAAR  and  must  be  fixed  for 
future  exercises. 

The  task  assessor  believed  that  the  AAR  tools  created  an  effective  structure  for  measuring 
performance,  collecting  evidence  and  understanding  the  decision  making  processes  of  the 
team  over  ti  me.  H  owever,  he  stated  that  hi  s  I  eadershi  p  of  the  A  A  R  was  affected  by  the  I  arge 
n  u  mber  of  tool  s  at  h  i  s  d  i  sposal  and  hi  s  fami  I  iarity  with  each  of  them.  O  n  some  occasi  ons  th  i  s 
led  him  to  make  use  of  a  tool  or  piece  of  information  thatwas  not  suited  to  the  purpose.  He 
also  believed  thathecould  haveprovided  moretargeted  feedback  if  hehad  been  aw  are  of  the 
affordances  of  each  tool  at  different  phases  of  thetrai  ni  ng  cycle.  This  issuemay  be  resolved  in 
future  exercises  if  the  assessor  is  trained  in  the  use  of  the  AAR  tools  prior  to  the 
commencement  of  the  exerci  se. 

The  task  assessor  indicated  that  the  requirements  of  an  AAR,  in  terms  of  the  primary  data 
source  to  be  displayed,  varied  across  individual  and  team  training  contexts.  Participants 
bel  i  eved  that  theP  en  si  eve  consol  e  record  i  ngs  w  ere  an  excel  I  ent  en  hancement  to  thetrai  n  i  ng  of 
individual  controllers,  but  in  a  team  training  context,  they  would  prefer  to  di  spl  ay  a  Soli  psys 
recordi  ng  rather  than  a  specifi  c  consol  evi  deo.  Thi  s  woul  d  have  provi  ded  an  i  nteracti  ve  vi  ew 
of  team  awareness,  that  is,  theability  to  pan  and  scan  the  map  display  and  ignore  individual 
user  interactions  with  the  interface.  It  is  not  yet  possibleto  synchronise  the  native  Solipsys 
recording  or  playback  with  TAARDIS  Console  ground  truth  logging  and  methods  of 
achi  evi  ng  thi  s  shoul  d  therefore  be  i  nvesti  gated . 

3.5.4  Suggested  AAR  tool  enhancements 

The  task  assessor  made  some  further  suggestions  on  how  theAAR  tools  might  be  enhanced. 
The  Solipsys  playback  functionality  could  be  extended  to  include  accompanying  selectable 
audio  recordi  ngs  taken  from  theoperator  stations.  This  should  enableswitching  between  the 
video  and  audio  recordi  ngs  as  appropriate.  This  may  be  added  to  a  multi-station  splitscreen 
recording  display,  which  would  be  useful  when  training  a  team  of  four  or  five  ABM's 
working  together  on  different  aspects  of  the  same  mission.  The  task  assessor  felt  that 
TAARDIS  and  ADGESIM  could  be  adapted  to  support  this  capability. 

3.5.5  Recommendations 

1.  The  task  assessor  found  it  difficult  to  concurrently  rate  performance  using  A  WAR  and 
create  ti  mestamped  bookmarks  using  TAARDIS.  The  effi  ci  ency  of  thi  s  process  wou  I  d  be 
i  mproved  if  task  assessors  were  asked  to  compl  ete  both  of  these  tasks  usings  WAR . 

2.  Additional  functionality  should  be  incorporated  into  the  A  WA R  tool  to  permit 
ti  mestamped  bookmarked  to  be  exported  to  other  applications  such  asTAARDIS. 

3.  JheTAARDIS  suite  of  tools  should  befurther  modified  to  automatethesynchronisation 
of  multiplevideo  and  audio  recordings. 
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4.  Assessors  should  be  adequately  trained  in  the  functionality  of  all  tools  and  procedures 
prior  to  the  commencement  of  an  exercise. 

5.  Technical  Staff  should  investigate  methods  of  synchronising  native  Soli  psys  recording  or 
playback  with  TAARDIS  Console  ground  truth  logging. 

6.  Technical  Staff  should  investigate  the  utility  of  extending  the  Solipsys  playback 
functionality  to  include  accompanying  selectable  audio  only  recordings  taken  from  the 
operator  stations. 


4.  Conclusions 

EBS08had  two  broad  goals.  These  were,  firstly,  to  examine  the  benefits  providing  a  RAAF 
ABM  team  with  a  synthetic  training  environment  within  which  to  prepare  for  their  tactical 
command  and  control  rol  e  i  n  P  B08,  and  second  I  y ,  to  provi  de  an  envi  ronment  w  i  thi  n  w  hi  ch  a 
number  of  new  simulation  tools,  decision  aids,  performance  measurement  tool  sand  training 
techniques  could  be  evaluated  and  developed  for  future  implementation  within  the  RAAF. 
The  focus  of  the  current  report  has  been  on  the  latter,  with  the  former  being  addressed  in  a 
companion  report  (Shanahan  et  al„  in  preparation).  The  exercise  provided  a  suitable 
envi  ronment  within  which  to  evaluate  an  evolved  envi  ronment  for  RAAF  collectivetraining 
composed  of  the  ADGESIM  suite  of  tools,  the  Toteboard,  AS  JETs-based  performance 
assessment  and  the  associated  AWAR  tool,  and  th  eTDT  framework.  The  body  of  this  report 
discusses  these  tools  and  techniques  in  terms  of  their  potential  for  use  in  a  RAAF  training 
context  and  suggests  methods  by  which  they  may  be  improved. 

Theversionof4  DGESIM  trialled  at  EBS08  was  considered  a  significant  improvement  over  the 
versi  on  cu rrentl  y  i  n  use  at  SA  CTU  and  SA  W .  The  RA  A  F  parti  ci  pants  w ere  i  mpressed  w  i  th  the 
potential  of  thenew  version  of  the4  DGESIM  system  to  run  effectively  with  fewer  peoplethan 
the  previ  ous  versi  on.  FI  owever,  dueto  ti  meconstrai  nts,  the  new  versi  on  of  A  DGESIM  was  not 
thoroughly  tested  prior  to  the  commencement  of  EBS08,  and  a  number  of  glitches  arose,  some 
of  w  hi  ch  d  i  sru  pted  the  exerci  se.  I  n  add  i  ti  on,  the  SI  M  0  Ps  felt  they  d  i  d  not  have  enough  ti  me  at 
thestartoftheexerciseto  becomefamiliar  with  thelatest  versi  on  of  A  DGESIM  ,  and  therefore 
did  not  understand  all  of  its  functionality.  All  efforts  will  bemadeto  provide  sufficient  time 
pri  or  to  the  commencement  of  future  exerci  ses  for  thorough  testi  ng  of  the  si  mu  I  ati  on  system 
and  operator  trai  ni  ng. 

TheABM  team  felt  that  the  Toteboard  provided  all  of  the  information  they  required  during 
mission  execution  and  provided  theadditional  benefit  of  facilitating  coordination  between  the 
tactical  control  centre  and  the  White  Force  control  centre.  The  RAAF  participants  offered 
suggestions  for  improvement,  which  have  si  nee  been  incorporated. 

TheABM  team  reported  that  all  four  of  theTDT  dimensions  were  familiar  constructs  that 
were  reflected  in  their  current  performance  assessment  but  were  more  implicitly  than 
explicitly  trained.  They  felt  that  theTDT  process,  as  experienced  at  EBS08,  added  I  ittleto  the 
development  of  teamwork  behaviours  within  the  ABM  team.  There  is  considerable  evidence 
to  suggest  that  training  that  focuses  on  team  processes  can  lead  to  improved  team 
performance  (Kozlowski  &  llgen,  2006).  There  is  also  considerable  support  for  the  TDT 


35 


DSTO-TR-2305 


framework,  as  an  effective  means  by  which  to  train  teams  to  work  together  better,  at  least 
within  the  context  of  the  US  Navy  (Smith-Jentsch,  etal.,  1998).  During  EBS08,  TDT  was  only 
partial  ly  implemented  and  therefore  strong  cond  usions  regard  i  ng  the  util  ity  of  the  approach 
cannot  be  drawn. 

The  AW AR  tool,  incorporating  the  ABM  DCA  team  goal  hierarchy  offered  a  marked 
improvement  on  current  RAAF  methodology  and  tools,  in  terms  of  the  breadth,  depth  and 
consistency  of  ratings.  Participants  felt  that  the  team  goal  hierarchy  would  also  be  useful  for 
designing  basic  course  scenarios  at  SACTU  and  SAW.TheABM  team  found  thevisual  layout 
of  the  debrief  user  interface  to  beintuitive.Thescoresand  comments  relating  to  each  sub-goal 
were  considered  extremely  useful,  as  they  provided  theteam  with  an  understanding  of  their 
current  performance,  showed  how  much  room  existed  for  improvement,  and  provided 
instructions  for  improving. 

The  RAAF  participants  found  the  AAR  tools  very  effective  in  minimising  the  amount  of 
preparati  on  requi  red  between  the  end  of  a  mi  ssi  on  and  the  begi  nni  ng  of  thetask  performance 
debrief.  Thetask  assessor  also  felt  that  the  AAR  tools  improved  the  time  efficiency  of  the 
debri  ef  process  itself  by  faci  I  i  tati  ng  the  brevity  and  sharpness  of  I  earni  ng  poi  nts.  The  pi  ayback 
of  video  and  audio  was  considered  an  extremely  useful  method  of  illustrating  important 
learning  points. 

Overall,  the  participants  found  EBS08to  bean  extremely  valuable  exercise  and  weresatisfied 
with  the  quality  of  the  preparati  on  they  recei  ved  for  PB08.  The  si  mul  ati  on  tool  s  and  trai  ni  ng 
techniques  evaluated  in  the  exercise  assisted  in  promoting  the  quality  of  this  experience  and 
prompted  participants  to  consider  methods  by  which  the  current  approach  of  the  RAAF  to 
training  could  beimproved.  Participants  provided  extensivefeedback on  each  tool  and  while 
some  shortcomi  ngs  w  ere  hi  ghl  i  ghted ,  val  uabl  e  suggesti  ons  for  i  mprovement  w  ere  obtai  ned . 
This  feed  back  w  i  1 1  gu  i  d  e  the  f  u  rther  d  evel  op  ment  of  these  tool  s  f  or  i  mp  I  ementati  on  i  n  f  utu  re 
research  exercises  and  in  RAAF  training  programs. 
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Appendix  A:  Continuum  of  simulated  training  events 


Event  type 

Purpose  of  training 
activity  and  relationship 
to  'live'  experiences 

Timing 

Mission 

phases 

Scenario 

management 

Simulation 

models 

Entity  control 

Scenario 

continuity 

Fidelity 

Doctrine 

D&selopment 

For  the  development  of 
future,  or  the  validation 
of  current,  tactics  and 
procedures  i  n  emergi  ng 
operational  contexts 

Not  in 
real¬ 
time 

Selected 

phases 

Experienced 

'game' 

controller 

required 

Highly 

abstracted 

models, 

limited 

parameters 

Course  entity 
behaviour,  scripted 
aggregate  entities 

Low 

Low 

Skills 

Development 

T o  enhance  the  ski  1  Is  of 
specific  categorisation(s), 
to  apply  the  current 
tactics  and  procedures 
within  generic 
operational  contexts 

Real¬ 

time 

Selected 

phases 

Distributed 

scenario 

management 

Basic  models, 
many 

parameters 

Variable  entity 
behaviour,  many  free 
play,  experienced  and 
rehearsed  role  players, 
one  role  player  to  one 
entity  group,  few 
scripted  entities 

Low 

Medium 

Exercise 

Workup 

To  f  ami  1  i  ari  se  sped  f  i  c 
sub-groups  or 
organisations  with  a 
specific  exercise 
environment,  and  refine 
plans  and  procedures  for 
its  conduct 

Real¬ 

time 

Most 

phases 

Central 

scenario 

management 

Representative 
models,  many 
parameters 

Variable  entity 
behaviour,  many  free 
play,  experienced  and 
rehearsed  role  players, 
one  role  player  to 
many  entity  groups, 
few  scripted  entities 

High 

High 

Mission 

Rehearsal 

To  improve  the 
performance  of  operators 
executing  a  prepared 
mission  plan 

Real¬ 

time 

All 

phases 

Central 

scenario 

management 

Highly 

detailed 

models, 

extensive 

parameters 

Fine  entity  behaviour, 
experienced  and 
rehearsed  role  players, 
one  role  player  to  one 
entity  group 

Continuous 

Very  high 

"a* 

T5 
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Appendix  B:  Summary  of  recommendations 

B.l.  ADGESIM  simulation  suite 

1.  All  simulator- related  technical  briefsshould  bedearand  contain  theappropriatelevel  of 
detail. 

2.  SIMOPs  should  receive  adequate  training  in  the  use  of  all  software  and  simulation 
systems  before  the  commencement  of  an  exerci  se.  The  SI  M  O  P  team  suggested  that  two 
days  of  focused  training  in  the  week  prior  to  EBS08  would  have  permitted  them  to 
become  sufficiently  familiar  with  the  new  versions  of  PSI  and  DISVOX . 

3.  Adequatetimeshould  be  reserved  in  the  weeks  preceding  an  exerci  sefor  a  thorough  and 
systemati  c  testing  of  ADGESIM  systems.  All  technical  issues  must  be  resolved  priortothe 
exerci  sewithno  ad  ditionsormodificationsmade  after  testi  n  g . 

4.  SIMOP  workload  will  beconsiderably  reduced  if  entitiesareassigned  by  role  rather  than 
location.  White  Force  scenario  managers  should  endeavour  to  achieve  this  wherever 
possible. 

5.  Further  workload  reduction  can  be  achieved  by  increasing  the  number  of  SIMOPs 
assigned  to  the  White  Force.  FI  owever,  all  qualified  and  partially  qualified  RAAF  SIMOPs 
were  present  at  EBS08  and  the  pool  of  avai  I  able  SI  M  OPs  must  therefore  be  increased. 

6.  Th  e  ADGESIM  simulation  suite  should  be  further  automated  with  the  goal  of  further 
reducing  the  number  of  SIMOPs  required  to  produce  a  complex  scenario.  FI  owever, 
modifications  must  be  sufficiently  tested  to  ensure  that  they  are  not  made  at  the  expense 
of  realism. 

7.  The  ABM  team  was  able  to  hear  the  activity  of  the  SIMOPs  and  adjusted  their 
communications  accordingly.  Ideally,  SIMOPs  and  ABM  teams  should  operate  in 
completely  separate  rooms  rather  than  partitioned  sections  of  the  same  room. 

B.2.  Toteboard 

1.  The  most  recent  version  of  the!  oteboard  should  beusedforall  futureABM  simulation 
exercises. 

B.3.  TDT  framework 

1.  If  a  TDT  introductory  exercise  is  used  to  train  a  team  to  recognise  instances  of  the  four 
TDT  elements,  careshould  betaken  when  selecting  the  scenario  description  to  ensure  that 
it  bears  relevance  to  the  context  within  which  the  team  usually  works.  The  description 
should  include  realistic  examples  of  good  and  bad  behaviour  as  well  as  extreme  examples 
included  to  maintain  interest.  Learning  to  recognise  these  subtle  examples  will  better 
equip  the  team  to  recognise  such  instances  in  their  own  performance. 
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2.  The  scenario  workload  must  be  high  enough  to  genuinely  test  teamwork  and 
communication  skills. 

3.  If  TDT  is  to  be  included  in  future  exercises,  assessment  should  becompleted  by  four  TDT 
assessors,  with  oneassessi  ng  performance  under  each  TDT  category.  If  this  is  not  possible 
due  to  personnel  constraints,  the  assessors  that  are  involved  should  undergo  extensive 
trai  ni  ng  i  n  TD  T  assessment  pri  or  to  the  commencement  of  the  exerci  se. 

4.  FutureTDT  assessors  should  undergotrainingthatimprovestheirunderstandingofthe 
importance  of  guiding  the  team  through  a  process  of  self-evaluation  and  correction, 
rather  than  providing  direct  and  unidirectional  feedback,  and  their  skills  in  encouraging 
team  member  participation. 

5.  The  ABM  team  reported  that  the  structure  inherent  to  TDT  did  not  compliment 
Australian  military  culture.  Modifications  aimed  at  improving  the  suitability  of  TDT 
w  i  thi  n  the  context  of  the  A  ustral  i  an  mi  I  i  tary  shoul  d  therefore  be  consi  dered . 

B.4.  AWAR  tool 

1.  Assessment  should  involve  the  evaluation  of  performance  on  a  pre-determined  list  of 
measures  that  covers  all  relevant  aspects  of  performance,  such  at  the  ABM  DCA  team 
goal  hierarchy. 

1.  Performance  ratings  will  be  more  objective  and  consistent  if  measures  refer  to  specific 
aspects  of  performance,  are  clearly  worded  and  are  unambiguous.  Although  the  ABM 
DCA  team  goal  hierarchy  offers  an  improvement  on  the  current  RAAF  assessment 
approach,  further  refi  nement  is  requi  red.  Further  research  wi  1 1  be  requi  red  to  i  nform  such 
improvements. 

2.  Structuring  training  scenarios  around  the  elements  of  the  team  goal  hierarchy  would 
promote  consistency  between  teams  in  terms  of  their  exposure  to  events  and  resultant 
learning. 

3.  Using  the  same  scenarios  for  all  students  at  a  particular  course  level  would  permit  reliable 
performance  comparisons  to  be  made  between  students. 

4.  Assessors  must  be  sufficiently  trained  in  the  use  of  tools  such  as  AWAR  and  the 
arrangement  of  elements  in  the  incorporated  hierarchy  before  using  them  to  assess 
performance. 

5.  Electronic  databases  of  student  performance  data  may  draw  attention  to  tasks  that 
students  find  particularly  difficult  and  training  may  be  modified  to  address  this.  This  and 
other  potential  uses  of  electronic  performance  data  should  be  considered. 

6.  Tools  like  AWAR  allow  performance  assessment  data  to  be  entered  straight  into  an 
el  ectroni  c  database,  ci  rcumventi  ng  thedoubl  e  hand  I  i  ng  that  currently  occurs  i  n  the  RAAF 
when  paper  and  pencil  performance  ratings  are  transferred  into  a  database. 
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7.  A  function  should  be  incorporated  into  performance  assessment  tools  that  permits  scores 
to  be  exported  to  a  spreadsheet  and  printed  in  a  user-specified  order.  This  performance 
report  should  include  three  to  five  good  aspects  of  their  performance  and  three  to  five 
suggestions  for  improvement  and  should  be  given  to  students  prior  to  the  debrief. 

8.  The  benefits  and  drawbacks  of  creating  a  process  for  students  to  usea  performance  report 
and  audio  and  video  replays  of  a  mission  to  debrief  themselves  should  be  considered. 

9.  Lightweight  tabid:  PCswith  handwriting  recognition  should  beconsidered  to  improvethe 
portability  of  software  tools  such  asAWAR.  Handwriting  recognition  software  should 
permit  assessors  to  store  commonly  used  symbols,  shorthand  and  abbreviations. 

10.  The4  1/1/ A  R  spreadsheet  should  provide  more  space  for  assessor  comments  and  should 
permit  freehand  diagrams  to  support  assessor  comments. 

B.5.  The  AAR  tools 

1.  The  task  assessor  found  it  difficult  to  concurrently  rate  performance  using  A  WAR  and 
created  mestamped  bookmarks  using  77MRD/S.Theefficiency  of  this  process  would  be 
i  mproved  if  task  assessors  were  asked  to  compl  ete  both  of  these  tasks  usings  WAR . 

2.  Additional  functionality  should  be  incorporated  into  the  AW AR  tool  to  permit 
ti  mestamped  bookmarked  to  be  exported  to  other  appl  icati  ons  such  asTAARDIS. 

3.  TheTAARDIS  suiteof  toolsshould  befurther  modified  to automatethesynchronisation 
of  multi plevideo  and  audio  recordings. 

4.  Assessors  should  be  adequately  trained  in  thefunctionality  of  all  tools  and  procedures 
prior  to  the  commencement  of  an  exercise. 

5.  Technical  Staffshould  i  nvesti  gate  methods  of  synchronising  native  Sol  ipsys  recording  or 
playback  with  TAARDIS  Console  ground  truth  logging. 

6.  Technical  Staff  should  investigate  the  utility  of  extending  the  Solipsys  playback 
functionality  to  include  accompanying  selectable  audio  only  recordings  taken  from  the 
operator  stations. 
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Appendix  C:  Semi -structured  interview  questions 

C .1.  ABM  instructor  (task  work  assessor)  questions: 

C.1.1  The  AAR  tools 

How  straightforward  and  timely  was  the  process  of  selecting,  seeking,  and 
synchronising  AAR  data  sources? 

-  H  ow  much  ti  me  do  you  need  to  prepare  the  data  sources  for  the  A  A  R? 

o  Did  you  find  thisto  beinhibitive?  Annoying?  Justified? 

Did  you  have  all  of  the  information  you  needed  to  make  the  intended  teaching 
points  clearly  and  effectively? 

o  Did  you  require  any  information  that  wasn't  available? 

-  Was  the  information  you  used  during  AARs  easily  accessible? 

-  Given  no  restrictions,  how  would  you  design  the  ideal  data  enhanced/  supported 
AAR? 

o  How  many  screens  would  you  have?  What  would  you  display,  etc.? 

Do  you  feel  that  the  information  avail  able  to  you  for  use  during  AARs  improved 
theABM  team's  understanding  of  your  comments? 

-  What  is  the  impact  on  learning  of  providing  students  with  objective  performance 
data? 

If  given  the  opportunity,  would  you  use  this  array  of  information  in  SACTU 
simulation  exercises? 

Isan  AAR  of  this  type  equally  suited  to  both  individual  and  team  training? 

Doyou  have  any  further  comments  on  the  use  of  additional  visual  data  to  support 
the  AAR. 

C.1.2  TheABM  DCA  Team  Goal  Heirarchy 

-  Can  you  comment  on  the  use  of  theABM  DCA  hierarchy  as  a  tool  for  scenario 
design? 

Doyou  feel  that  use  of  the  hierarchy  (or  something  similar)  would  add  anything  to 
the  current  RAAF  approach  to  designing  scenarios? 

-  Are  there  any  drawbacks  or  benefits  of  designing  scenarios  such  that  performance 
on  such  low  level  tasks  can  be  assessed? 

-  Doyou  value  the  current  approach  of  granting  SM  Es  the  flexibility  to  inject 
additional  training  experi ences  to  scenarios  in  real  time? 

-  Would  it  be  more  useful  to  assess  all  students  in  a  course  using  a  common 
scenario? 

-  Can  you  commentontheusefulnessofthehierarchyfor  performance  assessment? 
Do  you  feel  thattheuseof  this  hierarchy  (or  something  similar)  would  add 
anything  to  the  current  RAAF  approach  to  assessing  performance? 

Doestheteam  goal  hierarchy  promote  the  objectivity  of  performance  assessment? 
Does  the  hierarchy  cover  all  relevant  areas? 

o  A  re  there  any  tasks  that  are  i  important  to  assess  that  are  mi  ssi  ng? 

Did  the  hierarchy  prompt  you  to  assess  performance  i  n  areas  that  you  may  have 
otherw  i  se  forgotten? 
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Does  use  of  the  hierarchy  permit  a  val  id  assessment  of  performance? 

Does  the  use  of  A 1/1/  AR  populated  with  the  ABM  DCA  hierarchy  facilitate  a  more 
objective  measure  of  performance  than  is  currently  used  atSACTU? 
o  Is  this  beneficial? 

Did  use  of  the  hierarchy  makethe  process  of  assessment  unnecessarily  complex? 

-  A  re  there  any  benefits  or  drawbacks  to  bei  ng  assessi  ng  a  team  of  students  as 
opposed  to  an  individual? 

-  Can  you  comment  on  the  usefulness  of  the  hierarchy  for  performance  feed  back? 

Do  you  feel  thattheuseof  this  hierarchy  (or  something  similar)  would  add 
anything  to  the  current  RAAF  approach  to  providing  performance  feedback? 

Did  the  students  find  it  useful  during  theAAR  to  organise  thediscussion  of  their 
performance  around  theABM  task  hierarchy? 

-  Can  you  comment  any  further  on  the  use  of  theABM  DCA  hierarchy  for  scenario 
design,  performance  assessment  and  performance  feed  back? 

C.1.3  The  AW  A  R  tool 

-  A  re  there  any  benefits  or  drawbacks  of  usi  ng  the4  1/1/ A  R  tool  to  assess  team 
performance  rather  than  the  traditional  paper  and  pencil  assessment  method 
employed  atSACTU. 

How  did  you  find  the  organisation  of  measures? 

o  Could  you  find  the  required  measures  quickly  and  easily? 

H  ow  di d  you  fi nd  the  scori  ng  system? 

-  A  re  there  any  physical  limitationsthatrelatetotheuseof4  WAR? 

o  A  re  the  restrictions/!  1/1/  AR  poses  on  mobility  a  problem? 

-  What  do  you  feel  the  use  of  A 1/1/  AR  would  add  or  detract  from  assessment  if  it  were 
to  replace  current  assessment  techniques  at  SACTU  ? 

Does/\  1/1/  AR  provide  a  good  structure  for  theAAR? 

-  Was  it  useful  to  have  performance  data  available  in  an  electronic  format 
i  mmed i  atel y  after  the  mi  ssi  ons? 

How  did  theABM  team  find  the  organisation  of  thedebrief  user  i  nterf  ace? 
o  Was  it  intuitive? 

Did  the  students  find  the  aggregation  and  colour  coding  of  thedebrief  user 
interface  useful? 

Did  theABM  team  find  thegraph  of  average  performance  on  theASJETs  useful  for 
understanding  thei  r  performance  and  areas  for  improvement? 

-  If  given  the  opportunity  would  you  behappy  to  replace  the  assessment  methods 
currently  employed  atSACTU  with  theA l/l/A R  technique? 

-  Can  you  make  any  further  comments  on  the  use  of  AW  AR  for  performance 
assessment  and  feedback? 

C.2.  ABM  team 

C.2.1  TheAAR  tools 

Does  an  AAR  that  is  supported  by  these  different  sources  of  data  add  anything  to 
thedebrief  approach  currently  employed  by  the  RAAF? 

o  What  additional  information  can  be  obtained  from  thesedata  sources? 
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A  re  there  any  di  sadvantages  to  uti  I  i  si  ng  these  different  data  sources  i  n  debriefs 
when  compared  to  the  approach  currently  employed  by  the  RAAF? 

o  Are  there  any  features  of  your  normal  debrief  techniques  that  are  better  than 
the  setup  we  have  here  at  Black  Skies? 

o  Were  there  any  instances  where  thedata  display  was  more  distracting  than 
informative? 

Did  thedata  displayed  in  theAAR  increase  your  understanding  of  the  assessor's 
feedback  about  your  performance? 
o  How? 

Was  there  any  information  not  displayed  that  might  have  provided  useful 
feedback  on  your  performance? 

o  Would  you  I  ike  to  have  seen  statistics  such  as  shot  logs? 

If  there  were  no  restrictions,  how  would  you  design  the  ideal  data  display  to 
support  AAR? 

o  How  many  screens  would  you  have? 
o  What  information  would  you  display? 

How  easy  was  it  to  interpret  the  data  presented  in  theAAR,  for  example  the  source 
ID,  size,  clarity,  relative  positioning  of  information? 

Do  you  have  any  more  comments  on  the  utility  of  an  AAR  supported  by  these 
different  sources  of  data? 

2  TheABM  DCA  Team  Goal  Heirarchy 

In  what  ways  does  using  a  team  goal  hierarchy  differ  from  the  way  in  which  your 
performance  is  usually  assessed? 

o  Does  it  add  to  the  current  RAAF  approach?  How? 
o  Does  it  detract  from  the  current  approach?  How? 

Doyou  think  that  employing  a  team  goal  hierarchy  is  a  useful  way  to  structure  a 
team  performance  debrief? 

o  What  are  the  advantages  of  organising  theAAR  in  this  manner? 
o  What  arethe  disadvantages  of  having  such  a  tightly-focused  AAR? 

How  effectively  did  the  hierarchy  of  training  objectives  capture  all  the  tasks  you 
were  required  to  perform  as  a  team  during  the  EBS08  missions? 

o  What  aspects  of  your  task  were  not  captured  by  the  goal  hierarchy? 
TheTeamGoal  Hierarchy  contains  very  specific,  low-level  criteria  against  which 
performance  i  s  rated : 

o  What  arethe  benefits  associated  with  having  such  specific  criteria? 
o  I  n  what  ways  could  this  specificity  of  performance  criteria  detract  from  the 
feedback  you  receive  from  the  assessor? 

Does  using  a  team  goal  hierarchy  result  in  a  more  objective  assessment  of  your 
performance  than  the  methods  that  are  currently  used? 

Do  you  think  that  using  a  team  goal  hierarchy  is  more  likely  is  more  likely  to  result 
i  n  a  true  representation  of  your  performance  than  the  methods  that  are  currently 
used? 

What  arethe  benefits  of  being  assessed  as  a  team  rather  than  as  an  individual? 
What  are  the  drawbacks  to  bei  ng  assessed  as  a  team? 

Would  there  be  any  benefit  to  being  able  to  compare  the  performance  of  individual 
team  members  across  specific  objectives? 
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In  what  ways  would  this  focus  on  the  individual  enhance  the  training  benefit  of  the 
team  overall? 

Within  theTeam  Goal  Hierarchy,  therearevery  specific  performance  criteria, 
o  Would  you  have  preferred  to  have  been  given  a  more  comprehensive 
description  of  these  criteria  prior  to  the  first  mission? 
o  How  would  a  better  knowledge  of  these  specific  criteria  have  influenced  the 
way  you  approached  the  mission? 

Using  the  team  goal  hierarchy  allows  the  assessor  to  link  specific  mission  tasks 
with  high-level  Air  Force  goals: 

o  How  useful  is  itfor  you  to  beableto  seethe  relation  between  your  specific 
duties  and  these  high-level  Air  Force  tactical  goals? 
o  Do  you  thi  nk  that  thi  s  approach  to  i  dentifyi  ng  trai  ni  ng  goal s  -  associ  ati  ng 
specific  tasks  with  high  level  tactical  goals-  could  beimplemented  more 
widely  in  theAir  Force? 

Do  you  haveany  further  comments  on  the  ABM  DCA  team  goal  hierarchy? 

C.2.3  The  AW  A  R  tool 

-  Team  performance  on  each  specific  objective  in  theTeam  Goal  Hierarchy  was  rated 
on  a  5  point  scale  -  from  TERRIBLE  to  EXCELLENT.  How  effective  were  these 
ratings  in  providing  a  true  representation  of  the  performance  of  your  team? 

-  What  are  the  advantages  of  using  A 1/1/  AR  populated  with  theTeam  Goal  Hierarchy 
over  the  approach  to  performance  assessment  currently  employed  by  the  RAA  F? 

-  What  are  the  disadvantages  of  usings  1/1/  AR  populated  with  theTeam  Goal 

H  i  erarchy  over  the  approach  to  performance  assessment  currently  empl  oyed  by  the 
RAAF? 

Do  you  think  that  theA  1/1/  AR  tool  provides  a  good  structure  for  the  AAR? 
o  What  did  you  likeaboutA  1/1/  AR? 
o  What  did  you  not  like  about  A  WA  R? 

How  easy  was  itto  understand  the  organisation  of  the  material  displayed  during 
theAAR? 

o  What  aspects  of  the  material  displayed  were  not  easy  to  understand? 

-  AWAR  permits  performance  data  to  be  provided  in  an  electronic  format 
i  mmed  i  atel  y  fol  I  ow  i  ng  a  trai  n  i  ng  exerci  se  or  mi  ssi  on : 

o  Is  this  desirable? 

o  Does  immediate  feed  back  increase  the  training  benefit  of  an  exercise? 

I  n  theA  l/l/A  R  system,  the  scores  from  lower-level  measures  were  averaged  to  give  a 
score  for  higher-level  objectives.  How  valid  are  these  aggregate  scores  as 
performance  measures? 

-  The  ratings  of  you  performance  were  colour  coded.  Did  this  colour  coding  facilitate 
your  interpretation  of  the  performance  feed  back?  H  ow? 

In  theAAR,  a  graph  was  presented  which  summarised  the  assessor's  ratings  in  the 
six  broad  categories  of  training  objectives: 

o  Did  this  graph  increase  your  understanding  of  the  good  and  bad  aspects  of 
your  performance?  H  ow? 

o  Did  the  graph  help  you  to  understand  the  areas  in  which  you  need  to 
improve? 

How  could  A  l/l/A  R  be  modified  to  be  more  useful  or  effective  as  an  AAR  tool? 
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Would  you  be  happy  for  3CRU  to  implement  the  A 1/1/  AR  system  for  trai  ni  ng 
purposes? 

Do  you  have  any  further  comments  on  A  W  A  R  ? 

4  The TD T  intervention 

H  ow  easy  to  understand  werethefour  teamwork  concepts  (information  exchange, 
communication  delivery,  supporting  behaviour,  and  initiative  leadership)? 

o  Which  dimension  of  teamwork  is  most  relevantto  an  ABM  team? 

Were  you  already  familiar  with  these  teamwork  concepts? 

o  Does  theAir  Force  promote  theseteamwork  behaviours  in  a  less  formal 
way,  or  in  different  language? 

What  are  the  advantages  of  having  a  performance  review  focused  solely  on 
teamwork  processes? 

o  Arethereany  drawbacksto  having  a  team-process  specific  AAR? 

Do  you  thi  nk  that  there  i  s  a  place  for  a  more  overt  focus  on  teamwork  processes  i  n 
the  training  delivered  by  SACTU? 

o  How  do  you  think  trainees  at  SACTU  would  respond  to  TD  7-style 
performance  reviews? 

o  What  are  some  of  the  possible  learning  benefits  of  introducing  team- process 
training  at  SACTU? 

o  Whatdisadvantagesmighttherebetointroducingthiskind  ofteam- 
process  trai  ni  ng  at  SACTU  ? 

At  the  begi  nni  ng  of  the  week  we  gave  you  a  vi  gnette  about  the  fri  endl  y-fi  re 
incident  involving  the  US  Blackhawks,  and  asked  you  to  pick  out  instances  of  good 
and  bad  teamwork  behaviours.  How  useful  was  this  exercise  in  illustrating  the 
TDT  concept? 

DidtheTDI  briefing  and  Blackhawk  exercise  improve  your  ability  to  recognise 
good  and  bad  teamwork  behaviours  during  the  mission  you  controlled  today? 
What  kind  of  impact,  if  any,  did  a  greater  awareness  of  team  behaviours  haveon 
your  mission  performance? 

Following  every  EBS08  mission,  you  were  debriefed  on  both  your  task  work 
performance  and  teamwork  performance  using  the  TDT  approach. 

o  Did  you  find  the  summary  of  how  well  you  worked  together  as  ateam 
useful? 

o  Whatimpactdid  thisTDT  summary  haveon  your  performance  as  ateam  in 
subsequent  missions? 

o  How  well  werethetwo  separate  debriefs  integrated? 
o  In  what  way  could  we  improve  the  integration  of  these  two  types  of 
debrief? 

What  aspects  of  the  TD  T  will  you  carry  through  to  future  exercises? 
o  How  do  you  think  these  aspects  of  TD  T  will  improve  your  work 
performance? 

o  What  ki  nd  of  i  impact  do  you  thi  nk  TD  T -styl  e  assessment  and  feedback 
would  haveon  your  performance  as  members  of  an  ABM  team  in  a  live 
control  environment? 

Do  you  have  any  further  comments  on  TDT ? 
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C.2.5  The/A  DGESIM  suite  of  tools 

-  Did  red  and  blue  entities  fly  as  you  would  expect  in  terms  of  acceleration,  climb 
rate,  turn  rates,  etc.? 

-  Did  red  and  blue  groups  appear  to  hold  formation? 

-  Were  the  weapons  ranges  of  red  and  blue  aircraft  accurately  represented? 

o  Were  they  consistent,  realistic,  etc.? 
o  Were  there  any  problems  with  the  accuracy  of  missile  hits? 

-  What  was  the  effect  of  any  lack  of  real  ism  that  was  experienced? 

Did  you  find  comms  between  yourself  and  red  aircraft  pilots  realistic  in  terms  of 
radio  quality  (e.g„  degradation  with  increased  distance  from  radio  towers)? 
o  What  were  the  effects  of  this? 

Did  you  experi  ence  any  probl  ems  relating  to  the  number  of  aircraft  the  si  m 
operators  were  required  to  control,  e.g„  delays,  unresponsiveness,  different  pilots 
with  same  voice... 

How  realistic werethesensor  inputsto  Solypsis? 

-  Was  the  detection  and  tracking  performance  of  the  system  realistic? 

o  If  not,  what  was  odd? 
o  What  was  the  effect  of  this  lack  of  real  ism? 

Do  you  have  any  further  comments  on  ADGE  SIM  ? 

C.3.  SI MOPs  and  WFMD 

C.3.1  The  A  DGESIM  suite  of  tools 

-  How  many  formations  have  you  been  controlling  over  the  course  of  this  exercise 
(range,  average)? 

-  What  level  of  workload  have  you  experienced  at  EBS08? 

o  Hasyour  workload  changed  over  the  course  of  the  week?  How?  Why? 

-  What  problems  resulted  from  your  high  level  of  workload? 

o  What  effect  did  these  probl  ems  have? 

Did  red  and  blue  entities  fly  as  you  would  expect  in  terms  of  acceleration,  climb 
rate,  turn  rates,  etc.? 

o  Did  red  and  blue  groups  appear  to  hold  formation? 
o  Were  the  weapons  ranges  of  red  and  blue  aircraft  accurately  represented? 

Were  they  consistent,  realistic,  etc.? 
o  Were  there  any  problems  with  the  accuracy  of  missilehits? 
o  Do  you  have  any  further  comments  on  realism? 

How  did  you  find  the  organisation  of  the  i  nformati  on  you  had  avail  able  to  you? 
o  Did  you  have  access  to  all  of  the  information  you  needed? 
o  Did  you  find  it  easy  to  access  the  right  information  at  the  right  time? 

How  did  you  find  the  presentation  of  information? 

o  Was  the  information  organised  suchthatyou  were  ableto  transmit 
i  nformati  on  from  the  si  m  i  nterf  ace  to  the  ABM  team  over  the  rad  i  o  w  i  th 
minimal  translation? 

-  Th  eADGESIM  developers  attempted  to  organise  information  in  an  intuitiveway 
so  that  you  had  spare  cognitive  capacity  for  higher  level  tasks,  e.g.,  thinking  ahead? 

o  Did  this  work  asthey  intended? 
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o  Doyou  feel  thatyou  wereableto  control  more  aircraft  as  a  result  of  the 
organisation  of  information? 

o  Do  you  have  any  further  comments  on  information  management? 

How  did  you  find  the  general  layout  of  the  display? 

o  Did  you  find  the  colour  coding  of  fuel  states  useful? 
o  Did  you  find  the  colour  by  IFF  feature  useful? 
o  Did  you  find  the  callsign  colour  coding  useful? 
o  Do  you  feel  that  all  relevant  information  was  accessible? 
o  How  did  you  find  the  filtering  of  information? 

Do  you  have  any  further  comments  on  A  DEGESI M  ? 
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Appendix  D:  TDT  introductory  vignette 


Instructions:  Read  through  this  true  story  of  the  shooting  down  of  two  US  Army  helicopters 
by  friendly  fire  in  Iraq  and  identify  examples,  either  positive  or  negative,  of  behaviours 
related  to  the  four  teamwork  dimensions  of  INFORMATION  EXCHANGE, 
COMMUNICATION  DELIVERY,  SUPPORTING  BEHAVIOUR,  and 
LEADERSHIP/  INITIATIVE. 


O n  A  pri  1 14, 1994,  at  07:36,  a  U  SA  F  E-3  A  W A  C S  ai  rcraft  from  the  963rd  A  i  rborne  A  i  r  C ontrol 
Squad  ron  (based  at  Ti  nker  A  i  r  Force  Base,  O  kl  ahoma)  departed  I  nci  rl  i  k  A  i  r  Base  (A  B),  Tu  rkey 
as  the  I  ead  ai  rcraft  of  52  coal  iti  on  ai  r  mi  ssions  scheduled  for  that  day  i  n  support  of  Operati  on 
ProvideComfort(OPC).TheAWACS,  under  thecommand  of  Major  LawrenceTracy,  was  to 
provide  airbornethreat  warning  and  air  control  for  all  OPC  aircraft  during  its  time  aloft.  The 
AWACScrew  reported  on  station  at  its  assigned  surveillance  orbit  located  insideTurkeyjust 
north  of  the  northern  border  of  I  raq  at  08:45.  The  weather  that  day  was  fair  and  clear  over 
northern  I  raq. 

At  08:22,  two  U  .S.  A  rmy  U  H  -60  Bl  ackhaw  k  hel  i  copters  from  the  6th  Battal  i  on,  159th  A  vi  ati  on 
Regi  ment,  cal  I  ed  Eagl  e  FI  i  ght,  departed  Piringlik,  near  Di  yarbaki  r,  Turkey  headed  for  theO  PC 
military  coordination  center  (MCC)  located  in  Zakhu,  Iraq.  Both  helicopters  were  fitted  with 
external,  230-gallon  fuel  sponsons  mounted  below  each  side  door  with  each  sponson 
emblazoned  with  large  American  flags.  In  addition  to  the  flags  on  the  sponsons,  each 
helicopter  was  marked  with  American  flags  on  each  side  door,  on  the  nose,  and  on  the  belly. 

At  09:21,  the  Bl  ackhaw  ks  reported  thei  r  entry  i  nto  the  no  fl  y  zone  i  n  N  orthern  I  raq  by  rad  i  o  on 
the  en-route  frequency  to  the  A  WACS  en-route  control  I  er,  Li  eutenant  J  oseph  H  al  cl  i ,  and  then 
landed  six  minutes  later  at  the  MCC.  H  al  cl  i  and  his  superior  officer,  Captain  Jim  Wang,  the 
A  WACS  crew's  Senior  Di  rector,  added  "friendly  helicopter"  symbology  to  their  radar  scopes, 
noted  that  both  hel  i  copters  were  di  sp I  ayi  ng  I  dentifi  cati  on  fri  end  or  foe  (I  FF)  si  gnal  s,  and  then 
suspended  the  symbol  ogy  after  the  Bl  ackhaw  ks  d  i  sappeared  from  thei  r  scopes  upon  I  and  i  ng 
at  the  MCC  at  09: 24.  Although  the  helicopters  were  squawking  the  wrong  IFF  codefortheno 
fly  zone  (called  the  Tactical  Area  of  Responsibility  or  TAOR),  neither  Wang  nor  Halcli 
informed  them  of  that.  Wang  and  Halcli  also  neglected  to  direct  the  Bl  ackhaw  ks  to  begin 
using  theTAOR  radio  frequency  instead  of  the  en-route  frequency. 

At  the  MCC,  the  Blackhawks  picked  up  16  members  of  the  U  N  Provide  Comfort  coalition 
leadershipteamincludingfiveKurdish  civilians, threeTurkish,two  British,  and  oneFrench 
military  officers,  plus  five  U.S.  civilians  and  military  officials.  At  09:54,  the  helicopters 
departed  from  the  M  C  C  for  A  rbi  I ,  I  raq.  The  Bl  ackhaw  ks  reported  thei  r  departure,  fl  i  ght  route, 
and  destinations  by  radio  which  was  acknowledged  by  Halcli.  Halcli  reinitiated  thefriendly 
helicopter  track  on  his  scope.  Halcli  placed  symbology  on  his  radar  screen  to  show  the  two 
Blackhawk  tracks  and  notified  Wang  of  the  helicopters'  movement.  In  addition  to  Halcli's 
screen,  thefriendly  helicopter  symbology  was  visible  on  the  radar  screens  of  Wang,  Tracy, 
and  USAF  Major  Doug  Martin.  Martin  was  the  "Duke"  or  airborne  command  element  on  the 


53 


DSTO-TR-2305 


A  W  A  C  S,  mean  i  ng  th at  h e  w  as  a  rated  p i  I  ot  assi  g ned  to  th e  crew  to  en su  re th at  al  I  en gagement 
(combat)  mandates  were  adhered  to  and  executed  as  written. 

Enrouteto  Arbil,  at  10:12,  the  Blackhawks  entered  mountainous  terrain  and  their  radar 
retu  rns  d  i  sap  peared  from  the  A  W  A  C  S'  scopes.  C  aptai  n  D  i  erd  re  Bel  I ,  an  ai  r  su  rvei  1 1  ance  off  i  cer 
on  the  AWACS,  noticed  that  the  Blackhawks'  radar  returns  had  disappeared  and  sent  an 
electronic  "attention  arrow"  to  Wang's  scope.  Wang  took  no  action  and  the  large  blinking 
green  arrow  automatically  disappeared  from  his  screen  after  one  minute. 

Meanwhile,  at09:35,twoUSAF  F-15C  fighter  aircraft  from  the  53rd  Fighter  Squadron,  piloted 
byCaptain  EricWickson  and  Lieutenant  Colonel  Randy  W.  May, departed  IncirlikAB.  Their 
mission  was  to  perform  an  initial  fighter  sweep  of  theTAOR  to  clear  the  area  of  any  hostile 
ai  rcraft  prior  to  the  entry  of  coalition  forces.  Theair  tasking  order  (ATO)  that  was  supposed  to 
list  all  scheduled  coalition  aircraft  missions  for  that  day  and  which  the  two  pilots  reviewed 
before  takeoff,  mentioned  thattherewould  bell.  S.  Army  Blackhawk  helicopters  operating  in 
theTA O  R  that  day  but  d i  d n 't  I  i  st  takeoff  ti  mes  for  them.  At  10:15,  W i  ckson  rad i  oed  M  arti  n  on 
the  AWACS  and  asked  if  he  had  any  information  to  pass  to  them.  Martin  replied  in  the 
negative. 

At  10:20  Wickson,  the  F-15C  flight  lead,  reported  entering  northern  Iraq  to  the  AWACS 
controller  responsible  for  air  traffic  insidetheTAOR,  USAF  Lieutenant  Ricky  Wilson.  The 
TAOR  frequency  that  the  F-15s  were  usi  ng  was  different  than  the  en-route frequency  bei  ng 
used  by  the  two  Blackhawks.  Wilson,  however,  was  monitoring  both  frequencies  as  well  as 
being  ableto  see  both  Blackhawks  on  his  radar  scope  before  they  disappeared  at  10:12.  Wilson 
and  the  other  AWACS  crewmembers,  many  of  whom  were  monitoring  the  F-15s  radio 
frequency,didnotinformtheF-15sthatBlackhawkswerecurrentlyoperatingintheTAOR. 
At  10:21,  Wilson,  believing  that  the  Blackhawks  had  landed  again,  asked  Wang  if  he  could 
drop  the  friendly  helicopter  symbology  from  the  AWACS'  scopes  and  Wang  approved  the 
request.  An  AWACS  crew  instructor,  Captain  M  ark  Cathy,  who  was  on  the  mission  to  assist 
theAWACScrew  and  superviseWilson  on  this,  his  first  mission  into  theTAOR,  had  retired  to 
the  back  of  the  ai  rpl  ane  at  10:00  to  take  a  nap. 

At  10:22,  W i  ckson  reported  a  radar  contact  on  a  I  ow-fl  yi  ng,  si  ow-movi  ng  ai  rcraft  40  mi  I  es  (64 
km)  southeast  of  his  current  position.  Wilson  acknowledged  Wickson's  report  with  a  "clear 
there"  response,  meani  ng  that  he  had  no  radar  contacts  i  n  that  area.  U  nbeknownst  to  the  two 
F-15  pilots,  the  unidentified  aircraft  were  the  two  U.S.  Army  Blackhawks.  NeitherTracy  or 
WangspokeupatthispointtorequestthatAWACS  crew  members  attempt  to  i  d  enti  fy  th  e  F- 
15s  radar  contacts. 

Both  F-15  pilots  then  electronically  interrogated  the  radar  target  with  their  on-board  IFF 
systems  across  two  different  frequencies.  Their  IFF  systems  responded  negatively  to  the 
attempt  to  identify  the  contact  and  the  F-15s  moved  to  intercept  the  unidentified  aircraft. 
Intermittent  IFF  returns  from  the  Blackhawks  now  began  to  show  on  Wilson's  and  other 
AWACS  crewmember's  scopes  and  friendly  helicopter  symbology  reappeared  on  Wang's 
scope.  After  cl  osi  ng  to  20  mi  I  es  (32  km)  of  the  radar  contacts,  at  10:25  the  F-15s  agai  n  reported 
the  contact  to  the  A  W  A  CS  and  W  i  I  son  thi  s  ti  me  responded  that  he  now  had  a  radar  contact  at 
that  reported  location.  Although  the  Blackhawk  intermittent  radar  and  now  steady  IFF  returns 
on  theAWACS  scopes  were  in  the  same  location  as  the  unidentified  contacts  bei  ng  tracked  by 
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theF-15s,  noneoftheAWACS  controllers  advised  Wickson  or  May  that  the  contacts  that  they 
were  tracking  might  be  friendly  helicopters. 

The  two  F-15snow  initiated  avisual  identification  (VID)  pass  of  the  contact.  The  VI D  pass 
entailed  violating  one  of  O PC's  rules  of  engagement,  which  prohibited  fighter  aircraft  from 
operating  below  10,000feet.  Atthistimethetwo  Army  Blackhawkshad  entered  a  deep  valley 
and  were  cruising  at  a  speed  of  130  knots  about  200  feet  (60  m)  above  the  ground.  Wickson's 
VID  pass  was  conducted  at  a  speed  of  about  450  knots  (833  km/  h),  500 feet  (150  m)  aboveand 
1,000  feet  (300  m)  to  the  I  eft  of  the  hel  i  copters.  At  10:28  W  i  ckson  reported  "Tally  2  H  inds"  and 
then  passed  by  and  tothefrontof  the  two  blackhawks.  "Flind"  istheNorth  Atlantic  Treaty 
Organisation  (NATO)  designation  for  the  Mil  Mi -24  helicopter,  a  helicopter  that  the  Iraqi, 
Turkish,  and  Syrian  militaries  operated  and  was  usually  configured  with  armament  on  small, 
side-mounted  wings.  Wilson  responded  with  "Copy,  Hinds"  and  asked  Wang,  "Sir,  are  you 
listening  to  this?"  Wang  responded,  "Affirmative"  but  offered  no  further  guidance  or 
comments. 

May  then  conducted  his  own  VID  pass  about  1,500  feet  (500  m)  above  the  helicopters  and 
reported  /'Tally  2".  M  ay  I  ater  stated  to  a  U  SA  F  acci  dent  i  nvesti  gati  on  board  that  hi  s  'Tal  I  y  2" 
call  meant  that  he  saw  two  helicopters  but  didn't  mean  that  he  was  confirming  May's 
indentifi  cation  of  them  as  I  raqi  Flinds.  Neither  F-15  pilot  had  been  informed  thatU.S.  Army 
Bl ackhawks  participating  in  OPC  often  carried  auxiliary  fuel  tanks  mounted  on  wings  nor  had 
either  been  instructed  in  the  paint  scheme  that  Iraqi  Flind  helicopters  used,  light  brown  and 
desert  tan,  which  was  different  than  the  dark  brown/  green  color  used  by  the  Blackhawks. 
Wickson  later  stated  that, "/  had  no  doubt  when  I  looked  at  him  that  hewasa  H  ind...TheBlackhawk 
did  not  even  cross  my  mind." 

Following  their  VID  passes,  Wickson  and  May  circled  back  behind  the  helicopters 
approximately  10  miles  (16  km).  At  10:28,  Wickson  notified  theAWACSthatheand  May  were 
"engaged"  and  instructed  May  to  "arm  hot" .  At  10:30,  Wickson  fired  an  AIM -120 AM  RAAM 
mi  ssi  I  e  at  the  trai  I  hel  i  copter  from  a  range  of  about  fou r  nauti  cal  mi  I  es.  The  mi  ssi  I  e  hi t  and 
destroyed  thetrailing  helicopter  seconds  later.  In  response,  the  lead  Blackhawk  immediately 
turned  and  dived  for  lower  altitudein  an  apparent  attempt  to  evade  the  unexpected  attack. 
Shortl  y  thereafter,  M  ay  fi  red  an  A I M  -9  Si  dew  i  nder  mi  ssi  I  e  at  the  I  ead  hel  i  copter  from  a  range 
of  about  one  and  a  half  nautical  miles,  hitting  and  shooting  it  down  also.  All  26  people  on 
board  thetwo  Blackhawks  were  killed.  After  flying  over  thewreckage  of  thetwo  Blackhawks 
lying  burning  on  the  ground,  May  radioed  Wickson, "  Stick  a  fork  in  them,  they' redone" . 

"1994 Black  Flaw kshootd own  incident"  (n.d.). 
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